Nov 03 21:42:24.087268 ip-10-0-131-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 21:42:24.087280 ip-10-0-131-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 21:42:24.087286 ip-10-0-131-89 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 21:42:24.087522 ip-10-0-131-89 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 21:42:34.336801 ip-10-0-131-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 21:42:34.336817 ip-10-0-131-89 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f66d0108253546e8a9473f5da26ebb32 -- Nov 03 21:44:48.369731 ip-10-0-131-89 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:44:48.724713 ip-10-0-131-89 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:44:48.724713 ip-10-0-131-89 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:44:48.724713 ip-10-0-131-89 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:44:48.724713 ip-10-0-131-89 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729857 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729940 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729945 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729948 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729951 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729954 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.729946 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729957 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729960 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729963 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729966 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729970 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729973 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729975 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729978 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729980 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729983 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729985 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729989 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729993 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729996 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.729999 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730002 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730004 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730007 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730009 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.730190 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730012 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730015 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730017 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730020 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730022 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730025 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730027 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730030 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730033 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730035 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730040 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730043 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730045 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730048 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730051 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730053 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730056 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730059 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730062 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.730661 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730064 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730067 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730069 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730072 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730074 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730077 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730079 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730082 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730084 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730087 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730091 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730094 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730096 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730099 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730101 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730104 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730106 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730109 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730111 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730114 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.731128 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730116 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730118 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730121 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730124 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730127 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730129 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730132 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730134 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730137 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730139 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730142 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730144 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730147 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730149 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730152 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730155 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730159 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730161 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730165 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.731596 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730168 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730171 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730173 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730176 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730178 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730180 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730183 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730186 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730188 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730190 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730193 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730196 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730198 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730201 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730203 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730206 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730210 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730212 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730215 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730218 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.732139 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730222 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.730225 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731001 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731008 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731011 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731013 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731016 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731019 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731021 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731024 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731026 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731029 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731033 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731036 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731038 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731041 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731043 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731046 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731049 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731051 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.732663 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731054 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731057 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731059 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731064 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731067 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731069 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731071 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731074 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731077 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731080 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731082 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731085 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731088 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731091 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731093 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731096 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731098 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731101 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731103 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.733143 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731105 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731108 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731110 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731113 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731115 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731118 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731121 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731123 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731126 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731128 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731131 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731134 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731137 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731140 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731142 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731145 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731147 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731149 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731152 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731154 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.733641 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731157 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731159 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731162 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731164 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731167 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731170 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731173 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731175 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731178 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731180 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731183 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731185 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731188 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731190 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731193 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731196 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731199 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731201 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731204 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.734113 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731207 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731209 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731212 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731214 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731216 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731219 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731221 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731224 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731226 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731229 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731231 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731234 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731236 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731239 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731243 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731246 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731249 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731252 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731255 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.734578 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731259 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731263 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731266 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731270 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731273 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731276 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731279 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731281 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731284 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731287 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731860 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731866 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731870 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731873 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731876 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731878 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731881 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731884 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731888 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.735024 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731892 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731895 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731898 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731901 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731904 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731907 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731909 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731911 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731914 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731916 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731919 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731922 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731925 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731927 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731930 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731932 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731935 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731937 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731939 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731942 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.735488 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731944 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731947 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731949 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731952 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731956 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731964 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731966 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731969 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731971 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731974 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731976 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731979 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731981 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731983 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731986 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731989 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731991 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731994 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.735969 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731996 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.731999 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732002 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732004 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732007 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732009 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732012 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732014 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732017 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732020 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732022 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732024 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732027 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732029 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732031 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732034 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732037 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732040 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732042 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732044 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.736411 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732048 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732050 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732053 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732055 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732058 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732060 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732063 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732065 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732068 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732071 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732073 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732076 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732078 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732081 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732085 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732088 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732090 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732093 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732095 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.736921 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732097 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732100 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732102 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732105 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732107 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732109 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732112 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732114 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732117 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732119 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732122 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732124 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732127 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732129 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732132 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732135 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732137 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732140 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.732144 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733008 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:44:48.737366 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733018 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733025 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733031 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733040 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733043 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733047 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733051 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733057 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733060 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733063 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733066 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733069 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733072 2576 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733075 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733078 2576 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733081 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733083 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733087 2576 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733090 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733092 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733095 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733099 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733103 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:44:48.737857 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733106 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733108 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733111 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733114 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733118 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733121 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733124 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733129 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733133 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733136 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733139 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733142 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733145 2576 flags.go:64] FLAG: --enable-server="true" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733148 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733152 2576 flags.go:64] FLAG: --event-burst="100" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733156 2576 flags.go:64] FLAG: --event-qps="50" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733159 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733162 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733165 2576 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733169 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733172 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733175 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733178 2576 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733181 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733184 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:44:48.738380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733187 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733190 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733193 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733196 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733199 2576 flags.go:64] FLAG: --feature-gates="" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733202 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733205 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733208 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733211 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733214 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733217 2576 flags.go:64] FLAG: --help="false" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733220 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733224 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733227 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733231 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733234 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733238 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733241 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733243 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733246 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733249 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733252 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733255 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733258 2576 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:44:48.739007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733261 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733263 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733267 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733270 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733281 2576 flags.go:64] FLAG: --lock-file="" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733286 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733289 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733292 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733298 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733300 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733303 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733306 2576 flags.go:64] FLAG: --logging-format="text" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733309 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733312 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733315 2576 flags.go:64] FLAG: --manifest-url="" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733318 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733322 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733325 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733329 2576 flags.go:64] FLAG: --max-pods="110" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733332 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733335 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733338 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733343 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733346 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733349 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:44:48.739606 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733352 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733358 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733361 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733364 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733367 2576 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733370 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733373 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733376 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733379 2576 flags.go:64] FLAG: --port="10250" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733382 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733384 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0aabf75d51c8e2ab0" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733388 2576 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733390 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733393 2576 flags.go:64] FLAG: --register-node="true" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733396 2576 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733400 2576 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733403 2576 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733406 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733409 2576 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733412 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733415 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733418 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733421 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733424 2576 flags.go:64] FLAG: --runonce="false" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733426 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733429 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733432 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:44:48.740193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733435 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733438 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733442 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733446 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733449 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733452 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733467 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733470 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733473 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733476 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733479 2576 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733482 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733487 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733490 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733493 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733497 2576 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733500 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733502 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733505 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733508 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733511 2576 flags.go:64] FLAG: --v="2" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733515 2576 flags.go:64] FLAG: --version="false" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733519 2576 flags.go:64] FLAG: --vmodule="" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733523 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733526 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:44:48.740932 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733672 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733676 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733680 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733685 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733688 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733691 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733693 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733696 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733698 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733703 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733706 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733709 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733711 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733714 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733717 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733719 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733722 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733724 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733728 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.741555 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733731 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733733 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733736 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733738 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733741 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733743 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733746 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733749 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733751 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733754 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733756 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733759 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733762 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733764 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733767 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733769 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733772 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733775 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733777 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.741997 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733779 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733782 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733789 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733792 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733795 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733797 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733800 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733803 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733805 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733807 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733810 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733817 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733819 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733821 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733824 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733826 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733829 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733831 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.742443 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733834 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733836 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733839 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733841 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733844 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733846 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733849 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733851 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733853 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733856 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733858 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733861 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733863 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733866 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733868 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733872 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733874 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733877 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733880 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733883 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.743033 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733885 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733887 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733890 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733893 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733897 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733899 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733902 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733904 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733907 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733909 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733913 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733917 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733920 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733922 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733925 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733927 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733930 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733933 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733935 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.743521 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733938 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733940 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733943 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733945 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733948 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733950 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733953 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733955 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.733959 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.733985 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734025 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734030 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734033 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734036 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734038 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734041 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734044 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.743965 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734047 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734050 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734052 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734055 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734058 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734060 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734063 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734065 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734068 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734071 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734073 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734075 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734078 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734081 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734084 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734088 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734092 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734095 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734097 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.744360 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734100 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734102 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734105 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734107 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734110 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734114 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734116 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734119 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734122 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734125 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734128 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734130 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734133 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734137 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734140 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734144 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734147 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734150 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734153 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734155 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.744825 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734158 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734160 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734163 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734166 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734168 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734170 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734173 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734177 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734180 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734182 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734185 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734188 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734191 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734193 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734195 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734198 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734201 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734204 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.745304 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734207 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734209 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734212 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734214 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734217 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734220 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734223 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734226 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734228 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734231 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734233 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734236 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734238 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734241 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734243 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734246 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734248 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734250 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734253 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734255 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.745795 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734258 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734260 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734263 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734265 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734267 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734270 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734272 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734274 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734277 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734279 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734284 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734287 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734289 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734292 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734295 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734298 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734301 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734303 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734307 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.746254 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734309 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.734312 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741044 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741058 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741074 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741078 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741131 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741136 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741139 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741143 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741146 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741149 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741151 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741154 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741156 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741159 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741161 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741164 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741166 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.746747 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741168 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741171 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741173 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741176 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741179 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741182 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741185 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741187 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741190 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741192 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741195 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741198 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741201 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741203 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741206 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741208 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741211 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741213 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741216 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741218 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.747171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741221 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741223 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741226 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741228 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741231 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741233 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741235 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741238 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741240 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741243 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741245 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741247 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741250 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741253 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741255 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741258 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741261 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741263 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741266 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.747871 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741268 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741270 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741281 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741284 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741287 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741290 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741293 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741296 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741300 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741304 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741307 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741309 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741312 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741314 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741317 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741319 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741322 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741324 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.748611 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741327 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741329 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741332 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741334 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741337 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741340 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741342 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741345 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741348 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741350 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741353 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741355 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741358 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741361 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741365 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741368 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741371 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741374 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741376 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741379 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.749153 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741382 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741384 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741387 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741389 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741392 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741394 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741397 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741399 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741402 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741404 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741406 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741409 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741411 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741414 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741416 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741419 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.749701 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741444 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741587 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741592 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741596 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741598 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741602 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741604 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741607 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741609 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741611 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741614 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741616 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741619 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741621 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741624 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741626 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741630 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741633 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741635 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.750117 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741637 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741640 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741642 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741645 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741647 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741649 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741652 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741654 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741657 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741659 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741662 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741667 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741669 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741672 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741674 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741676 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741679 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741681 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741684 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.750629 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741686 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741689 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741691 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741693 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741696 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741698 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741701 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741703 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741705 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741708 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741711 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741714 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741716 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741718 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741720 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741723 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741725 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741727 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741731 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741734 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.751105 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741737 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741739 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741742 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741744 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741746 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741749 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741751 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741753 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741756 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741758 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741760 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741763 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741765 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741768 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741770 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741772 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741775 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741777 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741783 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.751581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741786 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741788 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741791 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741793 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741796 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741798 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741801 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741803 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741805 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741808 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741810 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741812 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741815 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741817 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741819 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741822 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741824 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741826 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741829 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.752045 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741831 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741835 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741837 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741839 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741844 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741846 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741849 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741851 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.741872 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741902 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741907 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741910 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741912 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741914 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741917 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:44:48.752501 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741919 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741922 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741925 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741928 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741930 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741933 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741935 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741937 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741940 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741942 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741945 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741947 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741950 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741952 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741954 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741957 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741960 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741963 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741966 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741968 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:44:48.752878 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741971 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741973 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741976 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741978 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741980 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741984 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741988 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741992 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741995 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.741997 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742000 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742003 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742005 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742008 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742011 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742013 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742016 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742018 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742021 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:44:48.753376 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742023 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742026 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742028 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742031 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742033 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742036 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742040 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742043 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742047 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742050 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742052 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742055 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742057 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742060 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742062 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742065 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742067 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742070 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742072 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:44:48.753822 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742078 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742081 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742083 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742085 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742088 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742090 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742093 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742096 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742098 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742101 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742103 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742106 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742108 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742113 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742117 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742120 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742123 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:44:48.754282 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742126 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742129 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742132 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742135 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742137 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742140 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742142 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742145 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742147 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742150 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742152 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742155 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742158 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742162 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742164 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742167 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742169 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742171 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742174 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742176 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:44:48.754730 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742179 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:48.742182 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.742230 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.744129 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.744917 2576 server.go:1014] "Starting client certificate rotation" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.745013 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.745057 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.746584 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:44:48.755181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.748790 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:44:48.760690 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.760675 2576 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:44:48.763931 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.763915 2576 log.go:25] "Validated CRI v1 image API" Nov 03 21:44:48.764819 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.764789 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:44:48.768865 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.768838 2576 fs.go:135] Filesystem UUIDs: map[6dcb8c05-0aa3-4a9e-b0cb-776dbb2ac6cb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e1f8bfd3-7374-4e38-b5b8-a1e289f306cf:/dev/nvme0n1p3] Nov 03 21:44:48.768938 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.768864 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 21:44:48.774595 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.774469 2576 manager.go:217] Machine: {Timestamp:2026-11-03 21:44:48.773081453 +0000 UTC m=+0.302025031 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3016271 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2256dbfba219644f14ae359e02c655 SystemUUID:ec2256db-fba2-1964-4f14-ae359e02c655 BootID:f66d0108-2535-46e8-a947-3f5da26ebb32 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:4c:38:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:4c:38:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:70:9d:d1:23:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:44:48.774595 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.774588 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:44:48.774768 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.774693 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:44:48.776681 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776657 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:44:48.776782 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776767 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:44:48.776826 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776682 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-89.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:44:48.776826 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776824 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:44:48.776905 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776831 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:44:48.776905 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776872 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:44:48.776905 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.776885 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:44:48.777576 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.777565 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:44:48.781186 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781175 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:44:48.781308 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781299 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:44:48.781352 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781333 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:44:48.781352 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781343 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:44:48.781418 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781354 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:44:48.781418 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781364 2576 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:44:48.781418 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781384 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:44:48.781825 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.781813 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:44:48.783883 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.783862 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:44:48.788046 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.788032 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:44:48.788830 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.788815 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:44:48.788902 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.788846 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:44:48.789001 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.788991 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:44:48.789056 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789006 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:44:48.789056 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789013 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:44:48.789056 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789023 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:44:48.789056 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789029 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:44:48.789056 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789035 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:44:48.789539 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789521 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:44:48.789570 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789543 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:44:48.789570 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789551 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:44:48.789570 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789557 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:44:48.789665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789571 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:44:48.789665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.789581 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:44:48.790102 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.790093 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:44:48.790102 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.790102 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:44:48.794264 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.794249 2576 server.go:1263] "Started kubelet" Nov 03 21:44:48.794329 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.794288 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:44:48.794437 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.794371 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:44:48.794497 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.794480 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:44:48.794980 ip-10-0-131-89 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:44:48.798902 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.798518 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:44:48.798902 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.798844 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-89.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 21:44:48.802005 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.801968 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:44:48.802470 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.802430 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 21:44:48.802580 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.802485 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-89.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 21:44:48.804096 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.804066 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 21:44:48.804207 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.804187 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bfjdv" Nov 03 21:44:48.805282 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.805258 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:44:48.805999 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.805983 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:44:48.806161 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.806142 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:48.806503 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806485 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:44:48.806503 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806488 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:44:48.806503 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806507 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:44:48.806671 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806613 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:44:48.806671 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806621 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:44:48.806903 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806889 2576 factory.go:55] Registering systemd factory Nov 03 21:44:48.806957 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.806906 2576 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:44:48.807181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.807167 2576 factory.go:153] Registering CRI-O factory Nov 03 21:44:48.807181 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.807183 2576 factory.go:223] Registration of the crio container factory successfully Nov 03 21:44:48.807299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.807230 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:44:48.807299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.807245 2576 factory.go:103] Registering Raw factory Nov 03 21:44:48.807299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.807255 2576 manager.go:1199] Started watching for new ooms in manager Nov 03 21:44:48.808119 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.808104 2576 manager.go:319] Starting recovery of all containers Nov 03 21:44:48.815133 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.815108 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-89.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 21:44:48.816082 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.815093 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-89.us-east-2.compute.internal.18e4a71ba629e5a9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-89.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-89.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 21:44:48.794019241 +0000 UTC m=+0.322962820,LastTimestamp:2026-11-03 21:44:48.794019241 +0000 UTC m=+0.322962820,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-89.us-east-2.compute.internal,}" Nov 03 21:44:48.817816 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.817800 2576 manager.go:324] Recovery completed Nov 03 21:44:48.818207 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.818191 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bfjdv" Nov 03 21:44:48.820736 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.820706 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 21:44:48.822350 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.822339 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:48.824110 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824095 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:48.824158 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824125 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:48.824158 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824139 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:48.824943 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824921 2576 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:44:48.824943 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824943 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:44:48.825058 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.824961 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:44:48.825176 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.825149 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:48.825984 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.825964 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:48.826073 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.826003 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:48.826073 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.826019 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:48.827170 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.827153 2576 policy_none.go:50] "Start" Nov 03 21:44:48.827170 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.827171 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:44:48.827284 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.827182 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:44:48.829227 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.829215 2576 policy_none.go:44] "Start" Nov 03 21:44:48.863043 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.863019 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:44:48.865238 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865220 2576 manager.go:341] "Starting Device Plugin manager" Nov 03 21:44:48.865310 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.865253 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:44:48.865310 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865264 2576 server.go:87] "Starting device plugin registration server" Nov 03 21:44:48.865531 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865520 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:44:48.865573 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865535 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:44:48.865665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865646 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:44:48.865738 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865729 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:44:48.865738 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.865738 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:44:48.866151 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.866133 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:44:48.866211 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.866178 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:48.966815 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.966676 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:48.968666 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.967774 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:48.968666 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.967800 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:48.968666 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.967811 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:48.968666 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.967838 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:48.976417 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:48.976382 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:48.976417 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.976403 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-89.us-east-2.compute.internal\": node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:48.992826 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:48.992804 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.020691 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.020665 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:44:49.022054 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.022039 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:44:49.022133 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.022059 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:44:49.022133 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.022084 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:44:49.022133 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.022118 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 21:44:49.027019 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.026999 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:44:49.093873 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.093837 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.123240 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.123201 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal"] Nov 03 21:44:49.123371 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.123319 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:49.124314 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.124298 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:49.124409 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.124327 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:49.124409 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.124337 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:49.126725 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.126713 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:49.126880 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.126865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.126927 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.126893 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:49.127486 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127448 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:49.127575 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127495 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:49.127575 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127505 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:49.127575 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127451 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:49.127575 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127572 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:49.127699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.127588 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:49.129574 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.129561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.129623 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.129583 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:44:49.130293 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.130280 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:44:49.130367 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.130298 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:44:49.130367 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.130334 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:44:49.158841 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.158823 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.161265 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.161251 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.194344 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.194322 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.208182 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.208158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.208239 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.208182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.208239 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.208199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.294976 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.294914 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.309231 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.309299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.309299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309251 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.309299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.309406 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.309406 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.309313 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.395379 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.395349 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.461879 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.461858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.464000 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.463986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.496256 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.496234 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.596865 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.596800 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.697311 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.697280 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.744794 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.744774 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 21:44:49.745193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.744908 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 21:44:49.798213 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.798186 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.823322 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.823277 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:39:49 +0000 UTC" deadline="2028-05-04 07:49:52.50985637 +0000 UTC" Nov 03 21:44:49.823322 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.823319 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13138h5m2.686541013s" Nov 03 21:44:49.898990 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:49.898919 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:44:49.903096 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.903084 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:44:49.907148 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.907123 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.910620 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.910603 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:44:49.918986 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.918970 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:44:49.920533 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.920519 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:44:49.928525 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:49.928510 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:44:50.087765 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:50.087732 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda387975d67fca887e2f3c912beb1760d.slice/crio-db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875 WatchSource:0}: Error finding container db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875: Status 404 returned error can't find the container with id db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875 Nov 03 21:44:50.088318 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:50.088294 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7d2a1c76cdd1f6aba5eb1c53d3f6b0.slice/crio-1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59 WatchSource:0}: Error finding container 1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59: Status 404 returned error can't find the container with id 1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59 Nov 03 21:44:50.092479 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.092446 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:44:50.304826 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.304753 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:44:50.596664 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.596588 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:44:50.782471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.782429 2576 apiserver.go:52] "Watching apiserver" Nov 03 21:44:50.792010 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.791981 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:44:50.792318 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.792296 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hvj6c","openshift-network-diagnostics/network-check-target-f6h64","kube-system/global-pull-secret-syncer-cw9gc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6plmd","kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb","openshift-cluster-node-tuning-operator/tuned-kdggf","kube-system/konnectivity-agent-hx8cd","openshift-multus/network-metrics-daemon-f9v7m","openshift-network-operator/iptables-alerter-thpfp","openshift-ovn-kubernetes/ovnkube-node-rxr85","openshift-dns/node-resolver-bnwgp","openshift-image-registry/node-ca-t265q"] Nov 03 21:44:50.794743 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.794713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.794855 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.794815 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:44:50.797033 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.797010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.799334 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.799312 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.799449 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.799375 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:44:50.801299 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.801279 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.801396 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.801347 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jbpmc\"" Nov 03 21:44:50.801396 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.801362 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.801689 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.801671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.805193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.805174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.807566 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.806990 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 21:44:50.808570 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.808049 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.808570 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.808182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.810023 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.810001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:50.813185 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.813165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.815644 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.815622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:50.815748 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.815691 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:44:50.815933 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.815911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.815947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.815977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgnq\" (UniqueName: \"kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pw5x\" (UniqueName: \"kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.816471 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tx5\" (UniqueName: \"kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnqvm\" (UniqueName: \"kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwf8\" (UniqueName: \"kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.817050 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.816781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.818346 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.818320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.821637 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.821618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:50.825192 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.823597 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:44:50.825192 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.824126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.844362 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.844341 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-675lw" Nov 03 21:44:50.857131 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.857107 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-675lw" Nov 03 21:44:50.907823 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.907803 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910117 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910135 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910149 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910175 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910212 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910242 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910306 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910310 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.910369 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.910324 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.911878 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.911858 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.912085 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.911971 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:44:50.912085 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912001 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.912085 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912037 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-txfgg\"" Nov 03 21:44:50.912085 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.911864 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lbb62\"" Nov 03 21:44:50.912085 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912076 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912130 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912164 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912207 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912209 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912227 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912169 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.912347 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.912269 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rdd92\"" Nov 03 21:44:50.913773 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.913713 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:44:50.913773 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.913738 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bc4mv\"" Nov 03 21:44:50.914784 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.914762 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:44:50.914890 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.914848 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hgwjm\"" Nov 03 21:44:50.914953 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.914904 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:44:50.914953 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.914950 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lqk2r\"" Nov 03 21:44:50.915052 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.914990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:44:50.915103 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.915066 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52n7c\"" Nov 03 21:44:50.915155 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.915132 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:44:50.915225 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.915203 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fc4bx\"" Nov 03 21:44:50.917793 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.917887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.917887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtq9p\" (UniqueName: \"kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:50.917887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.918068 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.918068 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.917938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.918177 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4dp\" (UniqueName: \"kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.918177 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918275 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918275 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.918275 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnqvm\" (UniqueName: \"kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.918275 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.918478 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.918478 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.918478 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918478 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.918699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918699 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwf8\" (UniqueName: \"kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.918939 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918932 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.918941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919037 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919457 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919207 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgnq\" (UniqueName: \"kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.919850 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.920430 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.919854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.920430 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920060 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.920430 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920110 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.920430 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.920652 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.920511 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:50.920652 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.920652 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.920578 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:51.420557135 +0000 UTC m=+2.949500722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:50.920652 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.920870 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920669 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.920870 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.920870 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:50.920870 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtfc\" (UniqueName: \"kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.921074 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.921074 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.920982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.921074 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.921224 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.921224 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.921224 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.921224 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.921261 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tx5\" (UniqueName: \"kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:50.921308 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:51.421293233 +0000 UTC m=+2.950236799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.921427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.921806 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921812 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.921973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw5x\" (UniqueName: \"kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.922273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbjc\" (UniqueName: \"kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.922805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.923184 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.923184 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.922859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.923184 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.923027 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.923184 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.923107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:50.924037 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.923977 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.924786 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.924731 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.933205 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.933177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwf8\" (UniqueName: \"kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:50.934286 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.934238 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgnq\" (UniqueName: \"kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:50.935197 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.935174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnqvm\" (UniqueName: \"kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:50.936074 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.935762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tx5\" (UniqueName: \"kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:50.937205 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:50.937177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw5x\" (UniqueName: \"kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:51.023091 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq9p\" (UniqueName: \"kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4dp\" (UniqueName: \"kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023269 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtfc\" (UniqueName: \"kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.023935 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.023994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.025180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbjc\" (UniqueName: \"kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024659 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.024941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.025185 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.025829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.025901 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.026665 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026836 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.026876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027286 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.027871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.027814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028326 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028047 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028326 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:51.028326 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028560 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.028600 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.028600 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028600 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028600 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.028721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028661 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.028721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.028721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.028721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028708 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.029104 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.029104 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.029104 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028793 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.029104 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.028809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.029434 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.029205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.029434 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.029324 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.029617 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.029595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.029866 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.029823 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.030222 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.030198 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:51.032323 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.032236 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerStarted","Data":"1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59"} Nov 03 21:44:51.035048 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.035022 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"a387975d67fca887e2f3c912beb1760d","Type":"ContainerStarted","Data":"db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875"} Nov 03 21:44:51.037023 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.037003 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:44:51.037126 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.037027 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:44:51.037126 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.037041 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:51.037126 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.037108 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:51.537089857 +0000 UTC m=+3.066033449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:51.038706 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.038688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq9p\" (UniqueName: \"kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.040320 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.040296 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4dp\" (UniqueName: \"kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.047466 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.047440 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbjc\" (UniqueName: \"kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.047707 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.047686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtfc\" (UniqueName: \"kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.112368 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.112285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:44:51.120801 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.120618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t265q" Nov 03 21:44:51.122863 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.122833 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9135178_a435_448e_b425_e821a0470160.slice/crio-79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f WatchSource:0}: Error finding container 79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f: Status 404 returned error can't find the container with id 79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f Nov 03 21:44:51.132453 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.132426 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9f0f1e_3e27_4f6a_aff3_a621b458030b.slice/crio-448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc WatchSource:0}: Error finding container 448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc: Status 404 returned error can't find the container with id 448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc Nov 03 21:44:51.146440 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.146410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:44:51.147552 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.147527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:44:51.151609 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.151578 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:51.157667 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.157600 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2076766_b393_4828_bf8f_30e8b64b22c4.slice/crio-de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00 WatchSource:0}: Error finding container de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00: Status 404 returned error can't find the container with id de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00 Nov 03 21:44:51.160865 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.160844 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4374a20c_4407_4bea_a0eb_d278dd8231b1.slice/crio-62cc0f096763e4b344e86f3e692328e795d3cfbee13f379747cbfa1f9996020e WatchSource:0}: Error finding container 62cc0f096763e4b344e86f3e692328e795d3cfbee13f379747cbfa1f9996020e: Status 404 returned error can't find the container with id 62cc0f096763e4b344e86f3e692328e795d3cfbee13f379747cbfa1f9996020e Nov 03 21:44:51.164100 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.164081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:44:51.164525 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.164480 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeec6a26_5dc3_4688_829d_f5316d6bda7d.slice/crio-5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c WatchSource:0}: Error finding container 5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c: Status 404 returned error can't find the container with id 5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c Nov 03 21:44:51.168087 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.167999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvj6c" Nov 03 21:44:51.178688 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.178666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:44:51.178798 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.178740 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647406a2_8819_4286_b814_eff7c4ec3930.slice/crio-49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6 WatchSource:0}: Error finding container 49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6: Status 404 returned error can't find the container with id 49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6 Nov 03 21:44:51.190581 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.189800 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb34c3d_2432_43b6_bfca_e172503a1bab.slice/crio-ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f WatchSource:0}: Error finding container ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f: Status 404 returned error can't find the container with id ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f Nov 03 21:44:51.201628 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.201604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:44:51.212737 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:44:51.212710 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefbfcec_8810_4b61_8cee_46091553ee7a.slice/crio-6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d WatchSource:0}: Error finding container 6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d: Status 404 returned error can't find the container with id 6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d Nov 03 21:44:51.428014 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.427939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:51.428014 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.427982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:51.428201 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.428111 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:51.428201 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.428113 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:51.428201 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.428168 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:52.428154407 +0000 UTC m=+3.957097975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:51.428201 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.428191 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:52.428184039 +0000 UTC m=+3.957127607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:51.630252 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.630132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:51.630415 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.630320 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:44:51.630415 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.630346 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:44:51.630415 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.630359 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:51.630606 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:51.630419 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:52.630399394 +0000 UTC m=+4.159342976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:51.861119 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.860444 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:39:51 +0000 UTC" deadline="2028-06-16 08:21:20.968132768 +0000 UTC" Nov 03 21:44:51.861119 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:51.860504 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14170h36m29.107632603s" Nov 03 21:44:52.041770 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.041369 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerStarted","Data":"de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00"} Nov 03 21:44:52.046258 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.046149 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnwgp" event={"ID":"a9135178-a435-448e-b425-e821a0470160","Type":"ContainerStarted","Data":"79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f"} Nov 03 21:44:52.053358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.053325 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d"} Nov 03 21:44:52.061731 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.061666 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thpfp" event={"ID":"7cb34c3d-2432-43b6-bfca-e172503a1bab","Type":"ContainerStarted","Data":"ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f"} Nov 03 21:44:52.069741 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.069323 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" event={"ID":"da3705e8-d005-4131-bc19-78c18fc31aa7","Type":"ContainerStarted","Data":"6219f4bcd5c03f46075a5f338725bd10c3846acc96318385218e652b55ee424f"} Nov 03 21:44:52.074825 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.074776 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kdggf" event={"ID":"4374a20c-4407-4bea-a0eb-d278dd8231b1","Type":"ContainerStarted","Data":"62cc0f096763e4b344e86f3e692328e795d3cfbee13f379747cbfa1f9996020e"} Nov 03 21:44:52.081422 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.081394 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t265q" event={"ID":"0c9f0f1e-3e27-4f6a-aff3-a621b458030b","Type":"ContainerStarted","Data":"448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc"} Nov 03 21:44:52.093677 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.093157 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"a387975d67fca887e2f3c912beb1760d","Type":"ContainerStarted","Data":"7fcdc75cdef329d15164e64abe2545777fa57c9796fe8033e5a729f56b6fddc4"} Nov 03 21:44:52.097040 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.097011 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvj6c" event={"ID":"647406a2-8819-4286-b814-eff7c4ec3930","Type":"ContainerStarted","Data":"49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6"} Nov 03 21:44:52.099068 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.099041 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hx8cd" event={"ID":"feec6a26-5dc3-4688-829d-f5316d6bda7d","Type":"ContainerStarted","Data":"5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c"} Nov 03 21:44:52.110950 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.110888 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" podStartSLOduration=2.11087115 podStartE2EDuration="2.11087115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:52.11040766 +0000 UTC m=+3.639351248" watchObservedRunningTime="2026-11-03 21:44:52.11087115 +0000 UTC m=+3.639814740" Nov 03 21:44:52.439093 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.438967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:52.439093 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.439071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:52.439298 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.439237 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:52.439352 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.439299 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:54.439281428 +0000 UTC m=+5.968225020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:52.439806 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.439760 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:52.439904 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.439814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:54.439797182 +0000 UTC m=+5.968740751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:52.641388 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.641321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:52.641638 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.641578 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:44:52.641638 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.641598 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:44:52.641638 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.641611 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:52.641805 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:52.641674 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:54.641655909 +0000 UTC m=+6.170599491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:52.861317 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.861230 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:39:51 +0000 UTC" deadline="2028-05-24 14:21:57.053877828 +0000 UTC" Nov 03 21:44:52.861317 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:52.861264 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13624h37m4.192616568s" Nov 03 21:44:53.024898 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:53.024870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:53.025063 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:53.024967 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:44:53.025338 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:53.025271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:53.025433 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:53.025398 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:44:53.025433 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:53.025423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:53.025596 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:53.025485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:44:53.118776 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:53.118520 2576 generic.go:356] "Generic (PLEG): container finished" podID="5d7d2a1c76cdd1f6aba5eb1c53d3f6b0" containerID="fc56af8042e0883965349e4c2769a9a24cba31d063efd20ca87ced3a2350f536" exitCode=0 Nov 03 21:44:53.118776 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:53.118671 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerDied","Data":"fc56af8042e0883965349e4c2769a9a24cba31d063efd20ca87ced3a2350f536"} Nov 03 21:44:54.131709 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:54.131664 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerStarted","Data":"aa002c79a54e1994bb292b1038aa8b81e30756d5d0db4aecc7b2d8db4e6a550e"} Nov 03 21:44:54.152903 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:54.152832 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" podStartSLOduration=4.152811035 podStartE2EDuration="4.152811035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:54.152027065 +0000 UTC m=+5.680970655" watchObservedRunningTime="2026-11-03 21:44:54.152811035 +0000 UTC m=+5.681754627" Nov 03 21:44:54.456912 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:54.456850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:54.457088 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:54.456950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:54.457202 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.457184 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:54.457278 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.457265 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:58.457244695 +0000 UTC m=+9.986188279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:54.457356 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.457337 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:54.457408 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.457373 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:58.457360771 +0000 UTC m=+9.986304353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:54.658829 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:54.658739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:54.659078 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.659058 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:44:54.659145 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.659087 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:44:54.659145 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.659101 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:54.659263 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:54.659178 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:44:58.659157527 +0000 UTC m=+10.188101096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:55.033982 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:55.033877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:55.034145 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:55.034084 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:44:55.034840 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:55.034816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:55.035270 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:55.034838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:55.035460 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:55.035433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:44:55.035844 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:55.035825 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:44:55.146207 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:55.146166 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="abf4c02918c4c412d2ca24381e9e148b4e31809742f90081afa7e1d0cc05c10c" exitCode=0 Nov 03 21:44:55.146705 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:55.146536 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"abf4c02918c4c412d2ca24381e9e148b4e31809742f90081afa7e1d0cc05c10c"} Nov 03 21:44:56.164219 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:56.164179 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hx8cd" event={"ID":"feec6a26-5dc3-4688-829d-f5316d6bda7d","Type":"ContainerStarted","Data":"aa45b09fd4479bd429c17c335915921b64397a0b4645e54903cda9718291a9da"} Nov 03 21:44:56.749999 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:56.749915 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:56.750879 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:56.750669 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:44:56.771603 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:56.770810 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hx8cd" podStartSLOduration=3.468494962 podStartE2EDuration="7.7707879s" totalImagesPullingTime="4.302292938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:56.184198897 +0000 UTC m=+7.713142492" watchObservedRunningTime="2026-11-03 21:44:56.7707879 +0000 UTC m=+8.299731497" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:57.023266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:57.023399 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:57.023616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:57.023749 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:57.023804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:57.025111 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:57.023876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:44:57.177453 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:57.176806 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t265q" event={"ID":"0c9f0f1e-3e27-4f6a-aff3-a621b458030b","Type":"ContainerStarted","Data":"95a5106ded987d825b2e1c9ec8c0a0a82fb5f595689ea5f4d61e929554d9ad33"} Nov 03 21:44:57.195551 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:57.195465 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t265q" podStartSLOduration=2.934639806 podStartE2EDuration="8.195444846s" totalImagesPullingTime="5.26080504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:57.19524103 +0000 UTC m=+8.724184624" watchObservedRunningTime="2026-11-03 21:44:57.195444846 +0000 UTC m=+8.724388438" Nov 03 21:44:58.181273 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.181229 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" event={"ID":"da3705e8-d005-4131-bc19-78c18fc31aa7","Type":"ContainerStarted","Data":"4a347956ee7b9a5ce1d5040b0cfbe4f532ef35e073e6b2c5883443a72d741fc5"} Nov 03 21:44:58.181785 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.181389 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:44:58.243716 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.243645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:58.411750 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.411714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:58.497927 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.497851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:58.497927 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.497915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:58.498781 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.498756 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:58.498877 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.498848 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:06.498825254 +0000 UTC m=+18.027768840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:44:58.500763 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.500663 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:58.500763 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.500733 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:06.500715215 +0000 UTC m=+18.029658791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:44:58.699660 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:58.699618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:58.699860 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.699798 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:44:58.699860 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.699819 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:44:58.699860 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.699832 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:58.700010 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:58.699893 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:06.699873586 +0000 UTC m=+18.228817171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.028350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.028380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:59.028502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:59.028621 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.028692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:44:59.028885 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:44:59.028785 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:44:59.122949 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.122919 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:44:59.189420 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.189385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" event={"ID":"da3705e8-d005-4131-bc19-78c18fc31aa7","Type":"ContainerStarted","Data":"17f1d9ecb5943d95a4134d5f7f5f13a81596fffe08d2d093573f7c7d23215944"} Nov 03 21:44:59.876346 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.876242 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:44:59.122943401Z","UUID":"ef53a02b-25cd-404e-9806-72735ba1def5","Handler":null,"Name":"","Endpoint":""} Nov 03 21:44:59.878871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.878091 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:44:59.878871 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:44:59.878260 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:45:00.193810 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.193530 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvj6c" event={"ID":"647406a2-8819-4286-b814-eff7c4ec3930","Type":"ContainerStarted","Data":"1610f2d0ff491e40126f322264c9068d6854ab87b35df55a119ff085e029a333"} Nov 03 21:45:00.196964 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.196931 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="aa49f16a7252536c8dbae478d0ebfec497c98c17c0383d035c6098cb20ef1559" exitCode=0 Nov 03 21:45:00.197290 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.197211 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"aa49f16a7252536c8dbae478d0ebfec497c98c17c0383d035c6098cb20ef1559"} Nov 03 21:45:00.200100 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.199611 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnwgp" event={"ID":"a9135178-a435-448e-b425-e821a0470160","Type":"ContainerStarted","Data":"6495e76137abde4d0977a1b61af31074f5359d730de0dab18286f181bb77b03e"} Nov 03 21:45:00.243836 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.243773 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bnwgp" podStartSLOduration=2.516641056 podStartE2EDuration="11.243755539s" totalImagesPullingTime="8.727114483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:00.242838584 +0000 UTC m=+11.771782171" watchObservedRunningTime="2026-11-03 21:45:00.243755539 +0000 UTC m=+11.772699127" Nov 03 21:45:00.244017 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:00.243908 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hvj6c" podStartSLOduration=3.310721481 podStartE2EDuration="11.243902622s" totalImagesPullingTime="7.933181141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:00.219689631 +0000 UTC m=+11.748633220" watchObservedRunningTime="2026-11-03 21:45:00.243902622 +0000 UTC m=+11.772846208" Nov 03 21:45:01.022427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.022397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:01.022586 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:01.022545 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:01.022920 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.022903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:01.023028 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:01.023007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:01.023151 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.023135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:01.023322 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:01.023199 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:01.056108 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.056087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:01.206427 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.206393 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="0ee0904e2224a2ecb8c4faa3fdd56aea474e4a88dbfcbfb4f7ec2fc0927c2e11" exitCode=0 Nov 03 21:45:01.206882 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.206496 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"0ee0904e2224a2ecb8c4faa3fdd56aea474e4a88dbfcbfb4f7ec2fc0927c2e11"} Nov 03 21:45:01.208974 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.208927 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thpfp" event={"ID":"7cb34c3d-2432-43b6-bfca-e172503a1bab","Type":"ContainerStarted","Data":"5c8d9073afeb0c39bc9ddedec76c0fe00f990a54fed384b7b03402b2f30f71a7"} Nov 03 21:45:01.212132 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.212100 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" event={"ID":"da3705e8-d005-4131-bc19-78c18fc31aa7","Type":"ContainerStarted","Data":"fc27b7022401845aca2d30be0942a444b4929930ccf5e458dc9a264d0436d5c7"} Nov 03 21:45:01.217670 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.217645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:01.301550 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.301488 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" podStartSLOduration=3.572148613 podStartE2EDuration="12.301474126s" totalImagesPullingTime="8.729325513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.300935803 +0000 UTC m=+12.829879391" watchObservedRunningTime="2026-11-03 21:45:01.301474126 +0000 UTC m=+12.830417712" Nov 03 21:45:01.321502 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.321424 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-thpfp" podStartSLOduration=3.881693824 podStartE2EDuration="12.321403674s" totalImagesPullingTime="8.43970985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.320901113 +0000 UTC m=+12.849844700" watchObservedRunningTime="2026-11-03 21:45:01.321403674 +0000 UTC m=+12.850347263" Nov 03 21:45:01.436430 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.436353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:01.587730 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:01.587692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:02.119228 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:02.119193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:02.216051 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:02.216015 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kdggf" event={"ID":"4374a20c-4407-4bea-a0eb-d278dd8231b1","Type":"ContainerStarted","Data":"14c7bcff45403117429863ed47b17430d0dd59cf6c30772faf20219949baa84f"} Nov 03 21:45:02.237560 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:02.237514 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kdggf" podStartSLOduration=2.407984101 podStartE2EDuration="13.237501117s" totalImagesPullingTime="10.829517016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.23696345 +0000 UTC m=+13.765907048" watchObservedRunningTime="2026-11-03 21:45:02.237501117 +0000 UTC m=+13.766444704" Nov 03 21:45:02.265674 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:02.265645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:02.955395 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:02.955311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:03.022439 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.022398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:03.022609 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:03.022535 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:03.022609 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.022411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:03.022725 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:03.022647 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:03.022725 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.022398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:03.022824 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:03.022725 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:03.126201 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.126163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:03.228077 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.227994 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="73e5bd4819d818f64ee06161f3834dbf3270b13a73884571783e01d7f65d543a" exitCode=0 Nov 03 21:45:03.228679 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.228085 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"73e5bd4819d818f64ee06161f3834dbf3270b13a73884571783e01d7f65d543a"} Nov 03 21:45:03.332500 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.332442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:03.483239 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.483157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:03.758164 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:03.758091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:04.156821 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:04.156512 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:45:04.156975 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:04.156893 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:45:04.157424 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:04.157401 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:45:04.550648 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:04.550547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:04.955410 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:04.955299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.023221 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.023186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:05.023221 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.023222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:05.023221 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.023224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:05.023515 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:05.023322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:05.023515 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:05.023423 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:05.023623 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:05.023522 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:05.190076 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.190030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.235859 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.235826 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerStarted","Data":"0eb18176a128b0b3ed391a584a8697b5ad2d2c4371a8965269fdafaefd39041c"} Nov 03 21:45:05.342736 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.342697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.492189 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.492109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.758095 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.758026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.907360 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:05.907330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:06.088739 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.088668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:06.239971 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.239949 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="0eb18176a128b0b3ed391a584a8697b5ad2d2c4371a8965269fdafaefd39041c" exitCode=0 Nov 03 21:45:06.240105 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.239982 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"0eb18176a128b0b3ed391a584a8697b5ad2d2c4371a8965269fdafaefd39041c"} Nov 03 21:45:06.561419 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.561248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:06.565228 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.565209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:06.565286 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.565241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:06.565338 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.565328 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:45:06.565370 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.565333 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:45:06.565401 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.565386 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:22.565359788 +0000 UTC m=+34.094303354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:45:06.565401 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.565398 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:22.565392674 +0000 UTC m=+34.094336239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:45:06.766899 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:06.766822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:06.767330 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.766943 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:45:06.767330 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.766958 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:45:06.767330 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.766967 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:45:06.767330 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:06.767011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:22.766999584 +0000 UTC m=+34.295943163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:45:07.009650 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.009609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:07.022818 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.022771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:07.022818 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.022794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:07.022910 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:07.022876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:07.022949 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.022907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:07.023077 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:07.022983 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:07.023077 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:07.023055 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:07.165170 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.165136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:07.245149 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245115 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"9de9aece7e847ee6a4e454d03c0b772b7c966a50a605dcab46699332d985d2e2"} Nov 03 21:45:07.245295 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245189 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"89d4d6d18325dedd6065f100f8f014fa1b469b0d89d50def029340bbd598db35"} Nov 03 21:45:07.245295 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245233 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"da850130c73bcdb087083c5cf3d9ecad1c1eb99247eb805f7e1608e0bf88c041"} Nov 03 21:45:07.245295 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245261 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"db93eaac702ee21aa064f1a3227575d8ca3bc4f4793e7351667b7b8a68cd7d31"} Nov 03 21:45:07.245295 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245293 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"b48cfcdd99c970e3997d87208f4ce9f943c5b99301a1661d94a1ba4bda0aef37"} Nov 03 21:45:07.245508 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.245338 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"a2b8947292ee03d26023ddce5d15f43b723ca2059cf929ea7542783b1bf330da"} Nov 03 21:45:07.248100 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.248078 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="889b3bb5431bd17f336bd9ac5ea055f0d60e658dd988e014de519c74b5800f07" exitCode=0 Nov 03 21:45:07.248183 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.248110 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerDied","Data":"889b3bb5431bd17f336bd9ac5ea055f0d60e658dd988e014de519c74b5800f07"} Nov 03 21:45:07.357208 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.357183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:07.756363 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:07.756287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:08.253818 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.253744 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6plmd" event={"ID":"f2076766-b393-4828-bf8f-30e8b64b22c4","Type":"ContainerStarted","Data":"fb9d584e02c00a38661b0303a23c8f595343839b3baa3460b212411b1efeab72"} Nov 03 21:45:08.282339 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.282299 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6plmd" podStartSLOduration=8.284534357 podStartE2EDuration="19.282287493s" totalImagesPullingTime="10.997753136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:08.280301465 +0000 UTC m=+19.809245051" watchObservedRunningTime="2026-11-03 21:45:08.282287493 +0000 UTC m=+19.811231080" Nov 03 21:45:08.560689 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.560617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:08.714004 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.713975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:08.873840 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.873766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:08.906162 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:08.906133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnwgp_a9135178-a435-448e-b425-e821a0470160/dns-node-resolver/0.log" Nov 03 21:45:09.023125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.023096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:09.023310 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:09.023192 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:09.023310 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.023250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:09.023422 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:09.023312 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:09.023422 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.023358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:09.023525 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:09.023444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:09.024965 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.024929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.176736 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.176650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.259296 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.259261 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"1ebd38f8ae173b910238effb9f1702307b7e1c2eb611a892ad26c24006290fd6"} Nov 03 21:45:09.359167 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.359119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.552676 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.552586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.706017 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.705989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t265q_0c9f0f1e-3e27-4f6a-aff3-a621b458030b/node-ca/0.log" Nov 03 21:45:09.706640 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.706616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.895106 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:09.895024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.049303 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:10.049269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.295666 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:10.295573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.447048 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:10.447008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.759737 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:10.759713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.961149 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:10.960917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:11.023177 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.023155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:11.023287 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.023255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:11.023287 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.023282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:11.023394 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:11.023253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:11.023394 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:11.023364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:11.023482 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:11.023434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:11.153083 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.153046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:11.266133 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.266060 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" event={"ID":"cefbfcec-8810-4b61-8cee-46091553ee7a","Type":"ContainerStarted","Data":"e037e9b69391890a05f3d05efd4dbd93ef137c00ac733e609bd6a2082344df83"} Nov 03 21:45:11.266352 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.266336 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:11.266426 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.266413 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:11.282244 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.282221 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:11.300721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.300673 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" podStartSLOduration=7.282828258 podStartE2EDuration="22.300661843s" totalImagesPullingTime="15.017833585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:11.299965636 +0000 UTC m=+22.828909222" watchObservedRunningTime="2026-11-03 21:45:11.300661843 +0000 UTC m=+22.829605428" Nov 03 21:45:11.358391 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.358344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:11.589698 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.589580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:11.746019 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.745993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:11.932024 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:11.931952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:12.085037 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.084996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:12.268951 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.268870 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:12.284310 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.284278 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:12.408065 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.408037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:12.566406 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.566316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:12.763594 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.763555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:12.955348 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:12.955242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:13.023183 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.023155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:13.023317 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.023196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:13.023317 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.023248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:13.023317 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:13.023256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:13.023493 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:13.023319 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:13.023493 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:13.023388 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:13.111002 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.110965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:13.259135 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.259067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:13.414116 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.414087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:13.565622 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:13.565558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:14.152583 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:14.152551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:14.354225 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:14.354190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:15.022744 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:15.022717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:15.022744 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:15.022749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:15.023179 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:15.022718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:15.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:15.022816 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:15.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:15.022930 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:15.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:15.023001 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:17.022657 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:17.022627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:17.023179 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:17.022682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:17.023179 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:17.022698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:17.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:17.022757 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:17.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:17.022857 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:17.023179 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:17.022915 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:18.300122 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:18.300085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:19.024537 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:19.024502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:19.024711 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:19.024584 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:19.024711 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:19.024617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:19.024711 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:19.024617 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:19.024711 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:19.024694 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:19.024845 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:19.024790 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:21.023249 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:21.023220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:21.023249 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:21.023246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:21.023733 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:21.023220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:21.023733 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:21.023330 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:21.023733 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:21.023427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:21.023733 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:21.023543 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:21.496407 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:21.496371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:22.146584 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:22.146547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:22.340598 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:22.340568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:22.576749 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:22.576661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:22.576749 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:22.576699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:22.576962 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.576793 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:45:22.576962 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.576815 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:45:22.576962 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.576840 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs podName:78b46919-59fa-4ba1-ad2e-d226cc365941 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:54.576828517 +0000 UTC m=+66.105772094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs") pod "network-metrics-daemon-f9v7m" (UID: "78b46919-59fa-4ba1-ad2e-d226cc365941") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:45:22.576962 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.576875 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret podName:82debff8-da8f-4dfa-980b-374e735bd8c4 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:54.576857937 +0000 UTC m=+66.105801519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret") pod "global-pull-secret-syncer-cw9gc" (UID: "82debff8-da8f-4dfa-980b-374e735bd8c4") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:45:22.777375 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:22.777339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:22.777558 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.777522 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:45:22.777558 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.777541 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:45:22.777558 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.777552 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vb2zq for pod openshift-network-diagnostics/network-check-target-f6h64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:45:22.777655 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:22.777606 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq podName:8e517e3a-a9eb-4811-99b4-1e24a4001f94 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:54.777589677 +0000 UTC m=+66.306533258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb2zq" (UniqueName: "kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq") pod "network-check-target-f6h64" (UID: "8e517e3a-a9eb-4811-99b4-1e24a4001f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:45:23.026230 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.026196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:23.026572 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.026545 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:23.026714 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.026322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:23.026915 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.026895 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:23.027005 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.026269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:23.027146 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.027129 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:23.511922 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.511886 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cw9gc"] Nov 03 21:45:23.512533 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.512023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:23.512533 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.512143 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:23.512664 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.512641 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6h64"] Nov 03 21:45:23.512749 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.512737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:23.512841 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.512824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:23.513728 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.513705 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f9v7m"] Nov 03 21:45:23.513818 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:23.513802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:23.513926 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:23.513906 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:24.600649 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:24.600614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:25.000180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.000153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:25.023088 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.023066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:25.023088 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.023083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:25.023242 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.023087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:25.023242 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:25.023163 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:25.023242 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:25.023233 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:25.023394 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:25.023305 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:25.197153 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.197124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:25.350041 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:25.350004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:27.023540 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:27.023270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:27.023944 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:27.023283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:27.023944 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:27.023655 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9v7m" podUID="78b46919-59fa-4ba1-ad2e-d226cc365941" Nov 03 21:45:27.023944 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:27.023341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:27.023944 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:27.023713 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw9gc" podUID="82debff8-da8f-4dfa-980b-374e735bd8c4" Nov 03 21:45:27.023944 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:27.023829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6h64" podUID="8e517e3a-a9eb-4811-99b4-1e24a4001f94" Nov 03 21:45:27.377815 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:27.377738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:28.721940 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.721916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnwgp_a9135178-a435-448e-b425-e821a0470160/dns-node-resolver/0.log" Nov 03 21:45:28.727676 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.727655 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeReady" Nov 03 21:45:28.727771 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.727760 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 21:45:28.777643 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.777582 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hvnmj"] Nov 03 21:45:28.807260 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.807238 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-snnch"] Nov 03 21:45:28.807376 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.807358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:28.813593 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.813567 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:45:28.813853 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.813824 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-92tpk\"" Nov 03 21:45:28.813954 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.813936 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:45:28.823007 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.822985 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hvnmj"] Nov 03 21:45:28.823109 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.823034 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-snnch"] Nov 03 21:45:28.823109 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.823071 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8j8zm"] Nov 03 21:45:28.823109 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.823095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.837055 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.837035 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8j8zm"] Nov 03 21:45:28.837137 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.837124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.914723 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.914738 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.914779 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.914967 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.915049 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:45:28.915358 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.915306 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wc5kn\"" Nov 03 21:45:28.915790 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.915396 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:45:28.915790 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.915420 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:45:28.915790 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.915616 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7bvj7\"" Nov 03 21:45:28.918098 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:28.918209 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:28.918209 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv59g\" (UniqueName: \"kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:28.918209 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.918322 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt86\" (UniqueName: \"kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:28.918322 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtfj\" (UniqueName: \"kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.918406 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.918443 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.918522 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:28.918566 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:28.918600 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:28.918566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.018897 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.018860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.018897 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.018897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.018921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.018990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019125 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.019387 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv59g\" (UniqueName: \"kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:29.019479 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019540 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.019597 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019597 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt86\" (UniqueName: \"kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.019694 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtfj\" (UniqueName: \"kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.019842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.019821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.020082 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.020052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.023097 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.023070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.023191 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.023157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.023360 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.023338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:29.026931 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.026907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:29.027022 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.027007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:29.027266 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.027183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:29.031160 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.031108 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:45:29.031796 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.031777 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5xcnq\"" Nov 03 21:45:29.032033 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032019 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtfj\" (UniqueName: \"kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.032212 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032194 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv59g\" (UniqueName: \"kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:29.032285 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt86\" (UniqueName: \"kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.032360 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032341 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t98zr\"" Nov 03 21:45:29.032425 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032389 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p72km\"" Nov 03 21:45:29.032425 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032405 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:45:29.032515 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032493 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:45:29.032565 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.032552 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:45:29.143847 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.143815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:29.167040 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.167014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:45:29.185555 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.185535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:45:29.271804 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.271075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:29.423915 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.423886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:29.495721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.495610 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8j8zm"] Nov 03 21:45:29.497338 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.497304 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hvnmj"] Nov 03 21:45:29.497487 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.497365 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-snnch"] Nov 03 21:45:29.500484 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:29.500445 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38585226_48bc_4637_88aa_ff1cf9b9fd78.slice/crio-2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814 WatchSource:0}: Error finding container 2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814: Status 404 returned error can't find the container with id 2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814 Nov 03 21:45:29.501171 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:29.501136 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a9e1cd_c054_453d_bd1f_fdaa0a35e700.slice/crio-406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67 WatchSource:0}: Error finding container 406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67: Status 404 returned error can't find the container with id 406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67 Nov 03 21:45:29.501811 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:29.501785 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12498367_0192_4402_bb17_bfe4f3432d3a.slice/crio-146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d WatchSource:0}: Error finding container 146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d: Status 404 returned error can't find the container with id 146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d Nov 03 21:45:29.721697 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:29.721563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t265q_0c9f0f1e-3e27-4f6a-aff3-a621b458030b/node-ca/0.log" Nov 03 21:45:30.184759 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.184424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.307815 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.307783 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8j8zm" event={"ID":"12498367-0192-4402-bb17-bfe4f3432d3a","Type":"ContainerStarted","Data":"146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d"} Nov 03 21:45:30.309380 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.309351 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-snnch" event={"ID":"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700","Type":"ContainerStarted","Data":"e9c3ac861931ba3f386ccc5bca3f39a4f07f29f66c39df0e6351249a108ce31b"} Nov 03 21:45:30.309505 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.309403 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-snnch" event={"ID":"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700","Type":"ContainerStarted","Data":"406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67"} Nov 03 21:45:30.310544 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.310523 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hvnmj" event={"ID":"38585226-48bc-4637-88aa-ff1cf9b9fd78","Type":"ContainerStarted","Data":"2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814"} Nov 03 21:45:30.334657 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.334623 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh"] Nov 03 21:45:30.358840 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.358804 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl"] Nov 03 21:45:30.360016 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.359989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.366404 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.366367 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 21:45:30.366900 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.366800 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:45:30.370008 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.369987 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 21:45:30.376967 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.376947 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh"] Nov 03 21:45:30.377091 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.376987 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl"] Nov 03 21:45:30.377091 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.377010 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fnlbr"] Nov 03 21:45:30.377197 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.377102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.394170 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.394110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.399639 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.399414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.438949 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.438873 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 21:45:30.438949 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.438943 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 21:45:30.439175 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.438974 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 21:45:30.439175 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.439152 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:45:30.439270 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.439180 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vr8v6\"" Nov 03 21:45:30.439270 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.439208 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9smcc\"" Nov 03 21:45:30.439270 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.439238 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5f4x\"" Nov 03 21:45:30.456516 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.456442 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 21:45:30.456516 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.456511 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:45:30.456759 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.456472 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 21:45:30.456759 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.456541 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:45:30.529365 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529365 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529584 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.529584 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjkb\" (UniqueName: \"kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529584 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529682 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.529682 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thj6p\" (UniqueName: \"kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.529682 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529777 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529777 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529777 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.529777 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2vp\" (UniqueName: \"kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.529887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.529887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.529887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.529887 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.530027 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.530027 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.529929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.555188 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.555158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.631069 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.631069 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thj6p\" (UniqueName: \"kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.631312 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631312 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631312 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631452 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631310 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631452 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.631578 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631578 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2vp\" (UniqueName: \"kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.631578 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631556 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631721 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.631821 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.631821 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.631821 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.631959 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.631959 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.631959 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.631933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.632113 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:30.632023 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Nov 03 21:45:30.632113 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:30.632084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls podName:33267669-fbef-4d65-a6f3-23b24f74ca7f nodeName:}" failed. No retries permitted until 2026-11-03 21:45:31.132065098 +0000 UTC m=+42.661008684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls") pod "node-exporter-fnlbr" (UID: "33267669-fbef-4d65-a6f3-23b24f74ca7f") : secret "node-exporter-tls" not found Nov 03 21:45:30.632113 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.632091 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjkb\" (UniqueName: \"kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.635868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.636141 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.636963 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.636696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.637779 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.637754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.638413 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.638382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.638944 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.638920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.639331 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.639314 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.639511 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.639493 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.639582 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.639561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.642837 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.642815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.650378 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.649470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2vp\" (UniqueName: \"kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.650500 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.650417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thj6p\" (UniqueName: \"kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.652784 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.652758 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjkb\" (UniqueName: \"kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:30.716493 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.716390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:45:30.720864 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.720836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.819165 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.818975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:45:30.925771 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:30.925712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.057319 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.057148 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh"] Nov 03 21:45:31.059973 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:31.059948 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaaf2d59_8e11_4be0_a9c6_4dd5efb74d85.slice/crio-1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740 WatchSource:0}: Error finding container 1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740: Status 404 returned error can't find the container with id 1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740 Nov 03 21:45:31.083143 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.083117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.125127 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.125095 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl"] Nov 03 21:45:31.128792 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:31.128754 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24414154_24ae_4506_bb36_49d64ba5fcef.slice/crio-bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0 WatchSource:0}: Error finding container bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0: Status 404 returned error can't find the container with id bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0 Nov 03 21:45:31.143142 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.143109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:31.146501 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.146475 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:31.316500 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.316404 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-snnch" event={"ID":"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700","Type":"ContainerStarted","Data":"f2a846fd102953831f0903730df9075fcac1e55b7e3d530cbe8f89aa0cef3c5b"} Nov 03 21:45:31.318928 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.318706 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" event={"ID":"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85","Type":"ContainerStarted","Data":"e08d1e0bd8abb97918e5eafae442868efd63563966efec2d8928a6a49db9c5f2"} Nov 03 21:45:31.318928 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.318792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" event={"ID":"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85","Type":"ContainerStarted","Data":"0fe53162219544f8a8f3d4dc7fed337133e8a48430225d7e04b179232db931e3"} Nov 03 21:45:31.318928 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.318824 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" event={"ID":"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85","Type":"ContainerStarted","Data":"1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740"} Nov 03 21:45:31.319217 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.318986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.320540 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.320485 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" event={"ID":"24414154-24ae-4506-bb36-49d64ba5fcef","Type":"ContainerStarted","Data":"bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0"} Nov 03 21:45:31.403224 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.403188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:45:31.432274 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:31.431940 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33267669_fbef_4d65_a6f3_23b24f74ca7f.slice/crio-4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249 WatchSource:0}: Error finding container 4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249: Status 404 returned error can't find the container with id 4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249 Nov 03 21:45:31.489750 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.489660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.645120 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.645049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.806474 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.806430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:31.963715 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:31.963679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:32.330517 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:32.330374 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hvnmj" event={"ID":"38585226-48bc-4637-88aa-ff1cf9b9fd78","Type":"ContainerStarted","Data":"5c70051fd82d0fa4e1b21c0adeff9057bfe90b914666e22de5f9def2d998e871"} Nov 03 21:45:32.330517 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:32.330445 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hvnmj" event={"ID":"38585226-48bc-4637-88aa-ff1cf9b9fd78","Type":"ContainerStarted","Data":"83ffa7e24d8ea99a1814f6e1f4acce64f7b23591c754552e964b96ccd27147e1"} Nov 03 21:45:32.331191 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:32.330687 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:32.338142 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:32.338113 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnlbr" event={"ID":"33267669-fbef-4d65-a6f3-23b24f74ca7f","Type":"ContainerStarted","Data":"4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249"} Nov 03 21:45:32.355257 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:32.355207 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hvnmj" podStartSLOduration=1.876263436 podStartE2EDuration="4.355192686s" totalImagesPullingTime="2.47892925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:32.354152092 +0000 UTC m=+43.883095682" watchObservedRunningTime="2026-11-03 21:45:32.355192686 +0000 UTC m=+43.884136273" Nov 03 21:45:33.126978 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.126931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:33.345370 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.345122 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-snnch" event={"ID":"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700","Type":"ContainerStarted","Data":"348b670543c868833ea0df17275225afc2b3432783732329a6820ab7950274e6"} Nov 03 21:45:33.348480 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.348427 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" event={"ID":"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85","Type":"ContainerStarted","Data":"959a13635deba7c1a6508017384fafd800c5573186108e61628098e4d4f86683"} Nov 03 21:45:33.351807 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.351777 2576 generic.go:356] "Generic (PLEG): container finished" podID="33267669-fbef-4d65-a6f3-23b24f74ca7f" containerID="80344c85b74671ddb3fcc4f050eacbab8f1794eb31ff9de58e46649303c36c14" exitCode=0 Nov 03 21:45:33.352926 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.352892 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnlbr" event={"ID":"33267669-fbef-4d65-a6f3-23b24f74ca7f","Type":"ContainerDied","Data":"80344c85b74671ddb3fcc4f050eacbab8f1794eb31ff9de58e46649303c36c14"} Nov 03 21:45:33.356849 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.356821 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" event={"ID":"24414154-24ae-4506-bb36-49d64ba5fcef","Type":"ContainerStarted","Data":"49498473a16fc0c910dbf1767f1bee4da92dd2f940054ba15e6715ab5c9dc4da"} Nov 03 21:45:33.356950 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.356886 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" event={"ID":"24414154-24ae-4506-bb36-49d64ba5fcef","Type":"ContainerStarted","Data":"70459ac9cf8fb3a0cd94bd31667646599af1a9d1edc96cf9021fc84b9d4383c7"} Nov 03 21:45:33.356950 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.356916 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" event={"ID":"24414154-24ae-4506-bb36-49d64ba5fcef","Type":"ContainerStarted","Data":"141dfa688da3759f61aab3c2dfc9ccc97ebfc27da5df82e82a2b37d3c3a67b07"} Nov 03 21:45:33.412926 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.412879 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-snnch" podStartSLOduration=2.145455464 podStartE2EDuration="5.412864395s" totalImagesPullingTime="3.267408931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:45:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:33.38878129 +0000 UTC m=+44.917724878" watchObservedRunningTime="2026-11-03 21:45:33.412864395 +0000 UTC m=+44.941807982" Nov 03 21:45:33.413139 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.413116 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" podStartSLOduration=1.704650602 podStartE2EDuration="3.413109709s" totalImagesPullingTime="1.708459107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:33.412530843 +0000 UTC m=+44.941474430" watchObservedRunningTime="2026-11-03 21:45:33.413109709 +0000 UTC m=+44.942053291" Nov 03 21:45:33.424756 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.424726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:33.462531 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.462482 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" podStartSLOduration=1.679450822 podStartE2EDuration="3.462436503s" totalImagesPullingTime="1.782985681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:33.461363128 +0000 UTC m=+44.990306716" watchObservedRunningTime="2026-11-03 21:45:33.462436503 +0000 UTC m=+44.991380106" Nov 03 21:45:33.578531 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.578490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:33.730335 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:33.730248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:34.364013 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.363969 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnlbr" event={"ID":"33267669-fbef-4d65-a6f3-23b24f74ca7f","Type":"ContainerStarted","Data":"678d320241440fd0377055229465791e9ee97c4880ec447bf8fa311a738d4371"} Nov 03 21:45:34.364573 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.364034 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnlbr" event={"ID":"33267669-fbef-4d65-a6f3-23b24f74ca7f","Type":"ContainerStarted","Data":"234baaf346a657e7577e28c8b86ed4ecc3abb618cd63c1271f33aa86e2b146b5"} Nov 03 21:45:34.365659 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.365618 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8j8zm" event={"ID":"12498367-0192-4402-bb17-bfe4f3432d3a","Type":"ContainerStarted","Data":"58cd77f88828f9df7e21ba3e884b6d57e6151de7fd44b2214b9b63a9969745d7"} Nov 03 21:45:34.388322 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.388279 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fnlbr" podStartSLOduration=2.698996003 podStartE2EDuration="4.388265869s" totalImagesPullingTime="1.54612484s" totalInitContainerRuntime="143.145026ms" isStatefulPod=true podCreationTimestamp="2026-11-03 21:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:34.387081906 +0000 UTC m=+45.916025493" watchObservedRunningTime="2026-11-03 21:45:34.388265869 +0000 UTC m=+45.917209449" Nov 03 21:45:34.409720 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.409677 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8j8zm" podStartSLOduration=1.7428822510000002 podStartE2EDuration="6.409660361s" totalImagesPullingTime="4.66677811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:34.409145556 +0000 UTC m=+45.938089144" watchObservedRunningTime="2026-11-03 21:45:34.409660361 +0000 UTC m=+45.938603948" Nov 03 21:45:34.747612 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.746978 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79b56c756-lg8dh"] Nov 03 21:45:34.752642 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.752617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.758697 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.758672 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 21:45:34.759869 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.759837 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 21:45:34.761205 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.761172 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79b56c756-lg8dh"] Nov 03 21:45:34.786187 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.786155 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 21:45:34.786328 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.786229 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 21:45:34.786328 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.786250 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nqlh5\"" Nov 03 21:45:34.786407 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.786374 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2g3ek12ouge4r\"" Nov 03 21:45:34.876560 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876715 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876715 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkm9r\" (UniqueName: \"kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876715 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876865 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876865 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.876865 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.876852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.916852 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.916803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:34.977634 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm9r\" (UniqueName: \"kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.977803 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.977786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.978126 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.978085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.978531 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.978508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.978764 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.978744 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.981643 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.981621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.981853 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.981835 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.981962 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.981942 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:34.990444 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:34.990419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm9r\" (UniqueName: \"kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:35.063152 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.063068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:35.072736 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.072705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.275222 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.275184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.339064 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.339000 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79b56c756-lg8dh"] Nov 03 21:45:35.341771 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:35.341748 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb89632_d017_4912_9bd3_ae92a2e187fe.slice/crio-bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd WatchSource:0}: Error finding container bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd: Status 404 returned error can't find the container with id bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd Nov 03 21:45:35.369733 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.369704 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" event={"ID":"cfb89632-d017-4912-9bd3-ae92a2e187fe","Type":"ContainerStarted","Data":"bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd"} Nov 03 21:45:35.443292 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.443256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.451201 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.451177 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:45:35.453514 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.453498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.460748 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460730 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 21:45:35.460842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460753 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 21:45:35.460842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460785 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 21:45:35.460842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460796 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 21:45:35.460842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460814 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 21:45:35.460842 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.460822 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 21:45:35.462315 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.462303 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 21:45:35.463536 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.463517 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nmvdk\"" Nov 03 21:45:35.466805 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.466789 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 21:45:35.468104 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.468084 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:45:35.582066 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582173 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582173 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582235 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582235 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582297 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582297 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.582354 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.582314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.623435 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.623368 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75f954c659-h7nqb"] Nov 03 21:45:35.625417 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.625397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.629682 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.629661 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 21:45:35.629852 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.629736 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 21:45:35.629852 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.629832 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wc92d\"" Nov 03 21:45:35.630019 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.629878 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 21:45:35.630019 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.629997 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 21:45:35.642637 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.642618 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f954c659-h7nqb"] Nov 03 21:45:35.661033 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.660990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 21:45:35.665105 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.665074 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 21:45:35.683353 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683429 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683429 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683429 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683617 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683673 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683725 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.683790 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.684749 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.684701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.684859 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.684817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.684859 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.684845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.684981 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.683766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.688690 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.685378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.688690 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.686977 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.689577 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.689558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.697313 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.697286 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.763888 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.763836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:35.785694 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.785661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.785866 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.785714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznpn\" (UniqueName: \"kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.785866 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.785745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.785866 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.785856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.786074 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.785991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.786180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.786097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.786180 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.786143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.786284 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.786233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.827867 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.827833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:35.887193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887193 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887588 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887705 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887705 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887781 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznpn\" (UniqueName: \"kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887781 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.887781 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.887769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.888640 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.888592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.888874 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.888846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.888874 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.888856 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.889954 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.889928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.890254 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.890233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.890762 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.890738 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.890856 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.890832 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.900950 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.900931 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznpn\" (UniqueName: \"kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.936862 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.936840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:45:35.976692 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:35.976661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:36.043197 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:36.043167 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:45:36.046560 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:36.046528 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74320bc_ea6d_4469_98e7_7440ba94cef2.slice/crio-37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea WatchSource:0}: Error finding container 37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea: Status 404 returned error can't find the container with id 37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea Nov 03 21:45:36.223008 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:36.222975 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f954c659-h7nqb"] Nov 03 21:45:36.226600 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:36.226572 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baffe5c_d923_43b8_a3d4_7d1edf10e744.slice/crio-afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258 WatchSource:0}: Error finding container afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258: Status 404 returned error can't find the container with id afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258 Nov 03 21:45:36.375529 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:36.375491 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" event={"ID":"5baffe5c-d923-43b8-a3d4-7d1edf10e744","Type":"ContainerStarted","Data":"afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258"} Nov 03 21:45:36.376924 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:36.376884 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c56dbb448-ftbq5" event={"ID":"a74320bc-ea6d-4469-98e7-7440ba94cef2","Type":"ContainerStarted","Data":"37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea"} Nov 03 21:45:36.743346 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:36.743305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:37.382816 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:37.382770 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" event={"ID":"cfb89632-d017-4912-9bd3-ae92a2e187fe","Type":"ContainerStarted","Data":"b2ec3b3a82cda7f095487470a5fe586f21059d00e8e402fb1f175c5a187db381"} Nov 03 21:45:37.405296 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:37.405247 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" podStartSLOduration=2.12894239 podStartE2EDuration="3.405228997s" totalImagesPullingTime="1.276286607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:37.403921761 +0000 UTC m=+48.932865349" watchObservedRunningTime="2026-11-03 21:45:37.405228997 +0000 UTC m=+48.934172587" Nov 03 21:45:38.387372 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:38.387289 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" event={"ID":"5baffe5c-d923-43b8-a3d4-7d1edf10e744","Type":"ContainerStarted","Data":"1ec4de2357d0f3050c1b10a2eda00fee6e1eb7aa8f037bad55dbfc83165feaa9"} Nov 03 21:45:39.394852 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:39.394814 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" event={"ID":"5baffe5c-d923-43b8-a3d4-7d1edf10e744","Type":"ContainerStarted","Data":"f93f7b89f0cabb5d40e186d2aa350df98962d58799583f74797d218079592739"} Nov 03 21:45:39.395343 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:39.395317 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" event={"ID":"5baffe5c-d923-43b8-a3d4-7d1edf10e744","Type":"ContainerStarted","Data":"6200f156cc134cda375dee92874db2bb39c672027571e78be46196e95355328f"} Nov 03 21:45:39.428279 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:39.428218 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" podStartSLOduration=1.6908202060000002 podStartE2EDuration="4.428198943s" totalImagesPullingTime="2.737378737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:39.426097392 +0000 UTC m=+50.955041002" watchObservedRunningTime="2026-11-03 21:45:39.428198943 +0000 UTC m=+50.957142524" Nov 03 21:45:40.321239 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:40.321194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:40.399314 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:40.399281 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c56dbb448-ftbq5" event={"ID":"a74320bc-ea6d-4469-98e7-7440ba94cef2","Type":"ContainerStarted","Data":"1395bb81eaa38bfd170eea5d9aa469a7e0c52c92c8672e1898b3299f74a7c5ce"} Nov 03 21:45:40.423466 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:40.423415 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c56dbb448-ftbq5" podStartSLOduration=1.784497307 podStartE2EDuration="5.423401215s" totalImagesPullingTime="3.638903908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:40.422745348 +0000 UTC m=+51.951688935" watchObservedRunningTime="2026-11-03 21:45:40.423401215 +0000 UTC m=+51.952344801" Nov 03 21:45:40.474298 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:40.474271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:41.985957 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:41.985925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:42.143280 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:42.143251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:42.359779 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:42.359713 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hvnmj" Nov 03 21:45:43.521097 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:43.521059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:43.701105 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:43.701067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:44.147263 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:44.147217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:44.283736 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:44.283711 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:45:44.299772 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:44.299749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:44.449698 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:44.449619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:45.764105 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:45.764070 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:45.764509 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:45.764128 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:45.768802 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:45.768781 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:46.422068 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:46.422040 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:45:47.113633 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:47.113592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:47.339253 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:47.339222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:49.977222 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:49.977191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:50.190891 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:50.190853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:50.732704 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:50.732672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:50.936290 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:50.936260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:51.157114 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:51.157090 2576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:45:52.069795 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:52.069758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.221843 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:52.221814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:52.380608 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:52.380531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:54.637521 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.637481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:54.637901 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.637580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:54.641988 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.641970 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:45:54.642053 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.641986 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:45:54.650385 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.650363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:54.650536 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.650515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:54.839553 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.839519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:54.842744 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.842725 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:45:54.852753 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.852736 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:45:54.862298 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.862269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:54.882248 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.882227 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5xcnq\"" Nov 03 21:45:54.889334 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.889294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:45:54.910738 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.910721 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t98zr\"" Nov 03 21:45:54.917858 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.917840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:45:54.924417 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.924399 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p72km\"" Nov 03 21:45:54.931532 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:54.931512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:55.063351 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.063323 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:55.063514 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.063373 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:45:55.166027 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.164226 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cw9gc"] Nov 03 21:45:55.169342 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:55.169318 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82debff8_da8f_4dfa_980b_374e735bd8c4.slice/crio-7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600 WatchSource:0}: Error finding container 7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600: Status 404 returned error can't find the container with id 7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600 Nov 03 21:45:55.205526 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.205506 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f9v7m"] Nov 03 21:45:55.207759 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:55.207735 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b46919_59fa_4ba1_ad2e_d226cc365941.slice/crio-153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7 WatchSource:0}: Error finding container 153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7: Status 404 returned error can't find the container with id 153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7 Nov 03 21:45:55.218911 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.218885 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6h64"] Nov 03 21:45:55.221821 ip-10-0-131-89 kubenswrapper[2576]: W1103 21:45:55.221800 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e517e3a_a9eb_4811_99b4_1e24a4001f94.slice/crio-af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951 WatchSource:0}: Error finding container af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951: Status 404 returned error can't find the container with id af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951 Nov 03 21:45:55.447210 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.447130 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6h64" event={"ID":"8e517e3a-a9eb-4811-99b4-1e24a4001f94","Type":"ContainerStarted","Data":"af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951"} Nov 03 21:45:55.448325 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.448293 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9v7m" event={"ID":"78b46919-59fa-4ba1-ad2e-d226cc365941","Type":"ContainerStarted","Data":"153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7"} Nov 03 21:45:55.449423 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.449403 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cw9gc" event={"ID":"82debff8-da8f-4dfa-980b-374e735bd8c4","Type":"ContainerStarted","Data":"7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600"} Nov 03 21:45:55.606843 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.606805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:55.783983 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:55.783827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:57.461184 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:57.461137 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9v7m" event={"ID":"78b46919-59fa-4ba1-ad2e-d226cc365941","Type":"ContainerStarted","Data":"e4d6a7bb6c7c88694561096bf192afc443540a1e6a9f4c6cadc6fa6bda4b5660"} Nov 03 21:45:57.461635 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:57.461212 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9v7m" event={"ID":"78b46919-59fa-4ba1-ad2e-d226cc365941","Type":"ContainerStarted","Data":"af415c66dee4b20f30e1f841af1c8e11d5ab58b1d2099f4ebe3cc302c4b91268"} Nov 03 21:45:57.482559 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:57.482497 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f9v7m" podStartSLOduration=67.099162398 podStartE2EDuration="1m8.482480013s" totalImagesPullingTime="1.383317615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:57.481416932 +0000 UTC m=+69.010360513" watchObservedRunningTime="2026-11-03 21:45:57.482480013 +0000 UTC m=+69.011423595" Nov 03 21:45:59.198564 ip-10-0-131-89 kubenswrapper[2576]: E1103 21:45:59.198495 2576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:45:59.472087 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:59.472000 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6h64" event={"ID":"8e517e3a-a9eb-4811-99b4-1e24a4001f94","Type":"ContainerStarted","Data":"f56fad7b01955cf649141363b493b15f52cbe163eea79191cc5f98f94615035a"} Nov 03 21:45:59.472247 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:59.472128 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:45:59.490410 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:45:59.490235 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f6h64" podStartSLOduration=66.964799863 podStartE2EDuration="1m10.490219291s" totalImagesPullingTime="3.525419428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:59.489814385 +0000 UTC m=+71.018757976" watchObservedRunningTime="2026-11-03 21:45:59.490219291 +0000 UTC m=+71.019162879" Nov 03 21:46:00.171429 ip-10-0-131-89 kubenswrapper[2576]: I1103 21:46:00.171349 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:46:00.171496 ip-10-0-131-89 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 21:46:00.180167 ip-10-0-131-89 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 21:46:00.180339 ip-10-0-131-89 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 21:46:00.180525 ip-10-0-131-89 systemd[1]: kubelet.service: Consumed 4.289s CPU time. Nov 03 21:46:00.194085 ip-10-0-131-89 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:46:00.254103 ip-10-0-131-89 kubenswrapper[5973]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:46:00.254103 ip-10-0-131-89 kubenswrapper[5973]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:46:00.254103 ip-10-0-131-89 kubenswrapper[5973]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:46:00.254103 ip-10-0-131-89 kubenswrapper[5973]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:46:00.256083 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256015 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.256083 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256084 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256087 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256091 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256094 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256097 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256100 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256103 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256106 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256108 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256110 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256113 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256116 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256118 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256121 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256124 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256126 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256129 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256132 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256135 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.256142 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256147 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256151 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256153 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256156 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256159 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256162 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256164 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256167 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256171 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256175 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256178 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256181 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256184 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256187 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256189 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256192 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256195 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256197 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256200 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.256625 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256202 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256205 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256207 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256209 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256212 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256215 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256218 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256220 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256223 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256226 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256229 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256231 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256234 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256236 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256246 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256249 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256252 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256254 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256257 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256260 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.257102 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256262 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256265 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256267 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256269 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256272 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256274 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256277 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256279 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256282 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256285 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256287 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256289 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256292 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256294 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256297 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256299 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256302 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256305 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256308 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256310 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.257599 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256313 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256315 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256318 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256320 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256323 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256325 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256328 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256336 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256339 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256341 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256344 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256346 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256349 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256351 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256353 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256356 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256359 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256362 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256364 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.258072 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256369 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256372 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256375 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256378 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256380 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256383 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256385 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.256388 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257148 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257157 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257160 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257163 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257167 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257170 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257172 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257175 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257177 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257180 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257183 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.258557 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257185 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257189 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257194 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257197 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257200 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257202 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257205 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257207 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257210 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257212 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257215 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257217 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257220 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257222 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257225 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257228 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257231 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257234 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257236 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257239 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.259053 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257241 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257244 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257246 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257249 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257251 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257254 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257256 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257259 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257261 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257263 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257266 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257268 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257271 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257273 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257276 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257278 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257281 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257283 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257286 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.259548 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257288 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257291 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257293 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257295 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257299 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257302 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257304 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257308 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257311 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257313 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257316 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257318 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257321 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257323 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257326 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257328 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257331 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257333 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257336 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.260114 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257338 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257341 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257343 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257346 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257349 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257352 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257354 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257357 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257359 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257362 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257364 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257367 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257369 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257372 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257374 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257377 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257379 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257381 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257384 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.260870 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257386 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257389 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257393 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257397 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257399 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257402 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257405 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257407 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257410 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257412 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257415 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257417 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257419 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257422 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257424 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257427 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257434 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257436 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257933 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.261333 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257939 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257942 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257944 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257947 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257950 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257952 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257955 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257958 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257960 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257962 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257965 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257967 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257970 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257972 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257975 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257978 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257980 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257983 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257986 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.261842 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257988 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257991 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257994 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257996 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.257999 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258001 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258004 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258006 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258009 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258011 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258014 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258017 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258019 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258022 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258024 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258026 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258029 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258031 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258034 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.262332 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258037 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258039 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258042 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258045 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258047 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258050 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258052 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258054 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258057 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258060 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258062 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258064 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258067 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258069 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258072 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258074 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258077 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258082 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258085 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.262813 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258088 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258091 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258094 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258097 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258099 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258102 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258104 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258106 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258109 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258111 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258114 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258116 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258118 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258121 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258124 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258126 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258129 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258131 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258134 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258136 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.263261 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258139 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258141 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258144 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258146 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258148 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258151 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258154 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258156 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258159 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258161 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258164 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258167 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258171 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258174 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258177 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258179 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258182 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258184 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258187 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.263757 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258189 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258192 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258194 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258196 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258199 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258201 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258203 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258207 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258210 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258300 5973 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258309 5973 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258318 5973 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258324 5973 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258331 5973 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258336 5973 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258341 5973 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258345 5973 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258348 5973 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258351 5973 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258354 5973 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258358 5973 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258361 5973 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:46:00.264217 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258363 5973 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258366 5973 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258369 5973 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258372 5973 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258375 5973 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258402 5973 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258406 5973 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258409 5973 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258412 5973 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258418 5973 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258422 5973 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258425 5973 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258428 5973 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258431 5973 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258434 5973 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258437 5973 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258440 5973 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258443 5973 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258446 5973 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258450 5973 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258466 5973 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258469 5973 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258472 5973 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258475 5973 flags.go:64] FLAG: --enable-server="true" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258478 5973 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:46:00.264757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258481 5973 flags.go:64] FLAG: --event-burst="100" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258484 5973 flags.go:64] FLAG: --event-qps="50" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258487 5973 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258491 5973 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258494 5973 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258502 5973 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258505 5973 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258508 5973 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258511 5973 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258514 5973 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258517 5973 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258520 5973 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258523 5973 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258526 5973 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258529 5973 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258531 5973 flags.go:64] FLAG: --feature-gates="" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258537 5973 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258540 5973 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258543 5973 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258546 5973 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258548 5973 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258552 5973 flags.go:64] FLAG: --help="false" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258555 5973 flags.go:64] FLAG: --hostname-override="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258558 5973 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:46:00.265371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258561 5973 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258564 5973 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258568 5973 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258571 5973 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258574 5973 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258577 5973 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258580 5973 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258583 5973 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258585 5973 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258588 5973 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258593 5973 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258596 5973 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258599 5973 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258602 5973 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258604 5973 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258607 5973 flags.go:64] FLAG: --lock-file="" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258610 5973 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258613 5973 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258615 5973 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258621 5973 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258623 5973 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258626 5973 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258629 5973 flags.go:64] FLAG: --logging-format="text" Nov 03 21:46:00.265978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258632 5973 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258636 5973 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258639 5973 flags.go:64] FLAG: --manifest-url="" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258642 5973 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258646 5973 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258649 5973 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258652 5973 flags.go:64] FLAG: --max-pods="110" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258655 5973 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258658 5973 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258661 5973 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258664 5973 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258667 5973 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258670 5973 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258673 5973 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258680 5973 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258683 5973 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258686 5973 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258689 5973 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258691 5973 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258696 5973 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258698 5973 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258701 5973 flags.go:64] FLAG: --port="10250" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258704 5973 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258707 5973 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0aabf75d51c8e2ab0" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258710 5973 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258712 5973 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:46:00.266680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258715 5973 flags.go:64] FLAG: --register-node="true" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258718 5973 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258722 5973 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258725 5973 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258727 5973 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258730 5973 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258734 5973 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258738 5973 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258741 5973 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258744 5973 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258747 5973 flags.go:64] FLAG: --runonce="false" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258750 5973 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258753 5973 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258756 5973 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258758 5973 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258761 5973 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258764 5973 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258767 5973 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258770 5973 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258774 5973 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258776 5973 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258779 5973 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258782 5973 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258785 5973 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258788 5973 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:46:00.267329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258792 5973 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258797 5973 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258800 5973 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258802 5973 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258807 5973 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258809 5973 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258812 5973 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258815 5973 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258818 5973 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258821 5973 flags.go:64] FLAG: --v="2" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258826 5973 flags.go:64] FLAG: --version="false" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258830 5973 flags.go:64] FLAG: --vmodule="" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258833 5973 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.258836 5973 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258969 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258972 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258976 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258978 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258981 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258983 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258986 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258988 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258991 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.267949 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258994 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.258998 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259002 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259005 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259008 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259010 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259013 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259016 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259018 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259022 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259024 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259027 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259030 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259032 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259035 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259038 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259040 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259042 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259045 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.268558 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259047 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259050 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259052 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259055 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259059 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259061 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259064 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259066 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259069 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259072 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259074 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259077 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259079 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259082 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259084 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259086 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259089 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259091 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259094 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259097 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.269184 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259099 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259102 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259105 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259108 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259111 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259113 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259116 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259120 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259124 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259126 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259129 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259132 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259134 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259137 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259140 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259142 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259146 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259149 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259151 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.269834 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259153 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259156 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259158 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259161 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259163 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259166 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259168 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259171 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259173 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259176 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259179 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259181 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259184 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259187 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259189 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259193 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259196 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259198 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259200 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.270358 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259203 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259205 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259208 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259210 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259213 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259215 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259218 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259220 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259223 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259225 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259229 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259232 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259234 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259237 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259239 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259242 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259244 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259247 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259250 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.271068 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259252 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.259278 5973 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259313 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259317 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259320 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259323 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259325 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259328 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259331 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259334 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259337 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259340 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259342 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259345 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259347 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259350 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.271933 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259352 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259355 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259359 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259362 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259366 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259370 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259373 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259375 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259379 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259382 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259384 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259387 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259389 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259392 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259394 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259397 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259399 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259401 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259404 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.272659 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259406 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259409 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259411 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259414 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259416 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259419 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259423 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259426 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259428 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259431 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259433 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259435 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259438 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259440 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259443 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259445 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259449 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259451 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259467 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259470 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.273246 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259472 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259476 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259479 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259481 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259484 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259487 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259489 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259491 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259494 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259496 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259499 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259501 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259504 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259506 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259508 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259511 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259513 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259516 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259520 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.273821 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259522 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259525 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259528 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259530 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259534 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259537 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259540 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259543 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259545 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259548 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259551 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259553 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259557 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259559 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259563 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259566 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259568 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259570 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259573 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.274399 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259576 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259578 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259581 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259584 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259586 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259589 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259591 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259594 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259596 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259598 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259601 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259604 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259608 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259611 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.259614 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.263829 5973 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.263844 5973 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.263862 5973 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.263867 5973 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263919 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.274884 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263923 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263926 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263929 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263932 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263934 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263938 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263940 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263943 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263945 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263948 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263951 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263953 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263956 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263958 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263961 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263964 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263967 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263970 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263972 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263975 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.275351 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263977 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263980 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263982 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263985 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263988 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263990 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263992 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263995 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.263998 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264001 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264003 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264006 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264009 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264011 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264014 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264016 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264018 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264021 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264024 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.275890 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264026 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264029 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264032 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264034 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264037 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264039 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264042 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264045 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264047 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264049 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264052 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264054 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264057 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264059 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264061 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264064 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264066 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264069 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264072 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264076 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.276561 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264079 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264083 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264087 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264091 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264093 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264096 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264099 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264101 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264103 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264106 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264108 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264111 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264113 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264116 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264118 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264121 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264123 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264126 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264128 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.277249 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264131 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264134 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264136 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264139 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264141 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264144 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264146 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264148 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264151 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264153 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264155 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264158 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264160 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264163 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264165 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264168 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264170 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264173 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264175 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.277775 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264177 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264180 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264182 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264185 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264187 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264189 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264192 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264194 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.264220 5973 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264360 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264365 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264369 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264372 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264375 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264378 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264381 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.278308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264383 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264386 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264388 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264391 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264393 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264396 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264398 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264400 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264403 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264405 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264408 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264411 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264413 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264416 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264418 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264421 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264423 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264425 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264428 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264431 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.279022 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264433 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264436 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264439 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264443 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264446 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264449 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264452 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264467 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264470 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264473 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264476 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264479 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264482 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264484 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264487 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264489 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264493 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264496 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.279664 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264499 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264502 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264505 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264508 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264511 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264513 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264516 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264518 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264520 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264523 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264525 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264528 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264530 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264532 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264535 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264538 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264540 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264542 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264545 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.280129 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264547 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264550 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264552 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264555 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264557 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264560 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264562 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264565 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264568 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264570 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264572 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264575 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264577 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264580 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264582 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264584 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264587 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264589 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264592 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264594 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.280577 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264596 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264599 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264601 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264604 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264606 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264608 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264611 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264613 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264616 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264618 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264620 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264623 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264625 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264630 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264633 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264635 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264637 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264640 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264643 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264645 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.281050 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264649 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264651 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.264674 5973 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264706 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264710 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264713 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264718 5973 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264721 5973 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264723 5973 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264726 5973 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264728 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264731 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264734 5973 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264736 5973 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264739 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264741 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:46:00.281538 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264743 5973 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264746 5973 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264749 5973 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264751 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264754 5973 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264756 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264759 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264762 5973 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264764 5973 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264785 5973 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264791 5973 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264794 5973 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264796 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264799 5973 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264801 5973 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264804 5973 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264806 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264809 5973 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264812 5973 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264814 5973 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:46:00.281917 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264817 5973 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264820 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264823 5973 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264826 5973 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264828 5973 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264831 5973 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264833 5973 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264836 5973 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264838 5973 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264841 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264843 5973 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264846 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264849 5973 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264851 5973 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264853 5973 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264856 5973 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264859 5973 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264861 5973 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264864 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264866 5973 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:46:00.282452 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264869 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264871 5973 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264875 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264878 5973 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264881 5973 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264883 5973 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264886 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264888 5973 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264891 5973 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264893 5973 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264896 5973 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264899 5973 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264901 5973 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264904 5973 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264907 5973 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264910 5973 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264913 5973 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264916 5973 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264918 5973 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264921 5973 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:46:00.282975 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264923 5973 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264926 5973 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264928 5973 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264931 5973 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264933 5973 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264935 5973 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264938 5973 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264940 5973 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264943 5973 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264947 5973 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264951 5973 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264954 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264956 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264959 5973 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264963 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264966 5973 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264968 5973 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264971 5973 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264973 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:46:00.283446 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264976 5973 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264978 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264981 5973 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264983 5973 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264988 5973 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264993 5973 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.264996 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265000 5973 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265003 5973 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265005 5973 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265008 5973 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265010 5973 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265013 5973 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:00.265015 5973 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.265063 5973 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266009 5973 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266068 5973 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266260 5973 server.go:1014] "Starting client certificate rotation" Nov 03 21:46:00.284046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266427 5973 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266534 5973 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:39:49 +0000 UTC" deadline="2028-05-03 07:48:14.439582006 +0000 UTC" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266603 5973 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13114h2m14.17298257s" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.266807 5973 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.268718 5973 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.270427 5973 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.272803 5973 log.go:25] "Validated CRI v1 image API" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.274010 5973 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:46:00.284617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.275854 5973 fs.go:135] Filesystem UUIDs: map[6dcb8c05-0aa3-4a9e-b0cb-776dbb2ac6cb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e1f8bfd3-7374-4e38-b5b8-a1e289f306cf:/dev/nvme0n1p3] Nov 03 21:46:00.284886 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.275873 5973 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6219f4bcd5c03f46075a5f338725bd10c3846acc96318385218e652b55ee424f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6219f4bcd5c03f46075a5f338725bd10c3846acc96318385218e652b55ee424f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600/userdata/shm major:0 minor:368 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9f0f1e-3e27-4f6a-aff3-a621b458030b/volumes/kubernetes.io~projected/kube-api-access-lgwf8:{mountpoint:/var/lib/kubelet/pods/0c9f0f1e-3e27-4f6a-aff3-a621b458030b/volumes/kubernetes.io~projected/kube-api-access-lgwf8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~projected/kube-api-access-fv59g:{mountpoint:/var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~projected/kube-api-access-fv59g major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~projected/kube-api-access-thj6p:{mountpoint:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~projected/kube-api-access-thj6p major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~projected/kube-api-access-pmjkb:{mountpoint:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~projected/kube-api-access-pmjkb major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~projected/kube-api-access-twt86:{mountpoint:/var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~projected/kube-api-access-twt86 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~projected/kube-api-access-h9tx5:{mountpoint:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~projected/kube-api-access-h9tx5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~projected/kube-api-access-sznpn:{mountpoint:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~projected/kube-api-access-sznpn major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/647406a2-8819-4286-b814-eff7c4ec3930/volumes/kubernetes.io~projected/kube-api-access-2k4dp:{mountpoint:/var/lib/kubelet/pods/647406a2-8819-4286-b814-eff7c4ec3930/volumes/kubernetes.io~projected/kube-api-access-2k4dp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~projected/kube-api-access-fcgnq:{mountpoint:/var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~projected/kube-api-access-fcgnq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~secret/metrics-certs major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb34c3d-2432-43b6-bfca-e172503a1bab/volumes/kubernetes.io~projected/kube-api-access-rtq9p:{mountpoint:/var/lib/kubelet/pods/7cb34c3d-2432-43b6-bfca-e172503a1bab/volumes/kubernetes.io~projected/kube-api-access-rtq9p major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82debff8-da8f-4dfa-980b-374e735bd8c4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/82debff8-da8f-4dfa-980b-374e735bd8c4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e517e3a-a9eb-4811-99b4-1e24a4001f94/volumes/kubernetes.io~projected/kube-api-access-vb2zq:{mountpoint:/var/lib/kubelet/pods/8e517e3a-a9eb-4811-99b4-1e24a4001f94/volumes/kubernetes.io~projected/kube-api-access-vb2zq major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~projected/kube-api-access-wq5z4:{mountpoint:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~projected/kube-api-access-wq5z4 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9135178-a435-448e-b425-e821a0470160/volumes/kubernetes.io~projected/kube-api-access-7pw5x:{mountpoint:/var/lib/kubelet/pods/a9135178-a435-448e-b425-e821a0470160/volumes/kubernetes.io~projected/kube-api-access-7pw5x major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~projected/kube-api-access-2f2vp:{mountpoint:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~projected/kube-api-access-2f2vp major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~projected/kube-api-access-fjtfc:{mountpoint:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~projected/kube-api-access-fjtfc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~projected/kube-api-access-rkm9r:{mountpoint:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~projected/kube-api-access-rkm9r major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~projected/kube-api-access-mbtfj:{mountpoint:/var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~projected/kube-api-access-mbtfj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3705e8-d005-4131-bc19-78c18fc31aa7/volumes/kubernetes.io~projected/kube-api-access-qwbjc:{mountpoint:/var/lib/kubelet/pods/da3705e8-d005-4131-bc19-78c18fc31aa7/volumes/kubernetes.io~projected/kube-api-access-qwbjc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2076766-b393-4828-bf8f-30e8b64b22c4/volumes/kubernetes.io~projected/kube-api-access-qnqvm:{mountpoint:/var/lib/kubelet/pods/f2076766-b393-4828-bf8f-30e8b64b22c4/volumes/kubernetes.io~projected/kube-api-access-qnqvm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feec6a26-5dc3-4688-829d-f5316d6bda7d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/feec6a26-5dc3-4688-829d-f5316d6bda7d/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a629685f228895af7c12cbc0e33d6955401c51bf787756d6d044ade84762c2a6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1265349df4360bdc400db7e55e812f158784b77b813458f50ca006b80cd1c546/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/29b24bbfc4019e44b4c94adbe41502618af8952ee21ad8ebcd83fe77811adc2c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/640a104d677e3b3599cd1ee6d196f2ccf1f86fae62cb75fd012a45bad0bc1b9a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/152bcf876b7d7a4f4614e7a2ee4c30c0b7f24e6c3250b56921daf425ed06a939/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ce3fe8b992d23de4b3351501ada55930ba846894558c5e9650919e3359f191f5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f7b005f15377812d71f7de84395d7a59a09ac0b225fcfc63e1f7f4c68301b0ab/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4f9dbdaf0eb60630173f0bd7dcfd40b4a4dd02004c00ac7937eb661129ae1733/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/27f05f75150ff6c5d283f64c5b0efba1a0ba9541ef9f117daadace449913b58f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/86d15e5205a08901b54f2cf9d697118e3274540d297579b491863948420a3d30/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/dc218f2bbde1a8e2124257310a53e5f4398c344a6b65d00e339808692e5b7c06/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/d8828c9e36fe16b2a4dac692e0cd4d914530bc8a322e10b2d854f7f24a13c90d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/220ca54a7202b07355f63951888e0b19755cc08ff01618309912f58cdb6a8ecd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4a233039600046977381c150c381e4ecd6a69bdeae70b081670a3fb0da997e86/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/070e3fb238fb09743f6f1ae5b7eefddd0c7b72f36769260bbb0fb00e8d90ac31/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/containers/storage/overlay/95e0e276d3ba83d4a73b87677633584b0f167628b6d08da97e17f1d504e76920/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6132efeb9403552397becfd7dab7f9acffb120838e868eb433281224907da62a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e491651c6325f92cf5729329915d57c298a297295e0a349e0601b766e6c951ee/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/151b09fec685998f6bec0a218f0017be4ae6cab679049bec097b952e968d7aee/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/eb5089e28c56452be262f65a70296323401fb717e6c462a8f406d3e73dd0afa8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/595b25ada2d07db148f236d95d774262a61dcb17ff221edd043cff64d719d088/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7b66a17eebc761465f890b3a1ea3fbd70af44f28a8a4d818a5674cd7c5ed5143/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/cd90f5525fa32c97e976cc9788d1e70e2dcbf5d5927b02d9fc29846b49adb0f6/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/cc732629ffa9d2c7ad49d86bd98a6e45ea4b14ad4986c2f04d7905781acca18d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/b3dace9c6d2c83c35916be947b434f16accf5ace01b416c4bef39c2b50cc288b/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/4ab5b1886042b862f66adc4faaa5ec0509eb2ef2da9314be9b21626a99aea770/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/f60b204ba6126d495e979ca1f203d36158fe0414dc7529112a57c9e533e2e393/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/d739551c4e731d07210e1114a16ed925dde02119fbef8f842fc1cba3a4859e42/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/48d40c6ae2ed3450513245e004749d50183134d221dde3b7c76509ccd06bcaae/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/04b9262ed64a56d2e567e23aa7b16d15c289ea22385335bd513f881067a23fd0/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/620e053188d9355e80e4e27f787b3a9c62549e17197da71e0c8caa1bad4ba0c1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/0aac8e7d1d8a60bf733f582d7aec376f17d51f338a82fe09ff3e858fb4264c10/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5b879d2846f6cd20b619721826a29c40b083a395a7192cba6a14731184403a03/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1da9cb64b8f42335a4aaa934c3d3f366ad99fafa0837d0219b48df7424902865/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/dda823dcd9f1159bfb7b29d6e4dc34f1bf863888bcf0dce2cc4d85d280771f69/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/57bffbd02930d4cee55c1d7c0ec8a64a7012917dd472d1e12425213d0abecfd7/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/f2bc2a33c7f02395b1536c3a3579425ca07bbf9c5f3251a1b7a57df9ffebb07e/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/4048f44fc3de2f23392589111b4dbcabbba114e447d4923f8126342019bc63c2/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/54731e62520ed55ea59b6ba6dfba7faeffa4862c04281f06da9c4c16a9311834/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/2d4deef1aa0dfc41123da6595d8fd3c4221e9ca9f01037e9bf4ea05dc1abe95c/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/597f4dae35fc6cd90fc015973bda0338f0ee76b37c10eac15b17a2610e5265f1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/be3e5ee6c40c68407b7d76d410b99afd8a8af149f6e30b1b46f1438a9cd05b45/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/92cc99a487e3a1ced347ba73c4ccb8eadc81f7c869684ffceb542f60f1f42dd0/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/5a2899a9adf912e84e238926a316be3e264bdf73b4504e52c5523eff4b49bbe5/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/56db888c9915d1397ee2547fafbb7907219773c5f3976b914d6cf286f8309464/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/3e3c851926dd6bfb01d0801f438edb7bb1bca7e6b2cc5a9ab2e30bc89fd488ae/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/2c191823f33e4f97e0e7d7b20591bad7e8f7277f0dd9b6f029099c61664f3382/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/4465604d88deaec0eee36d068ef2674c5c53358e351a2791f43aeb277f969483/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/c4f8f2bce05b1397b8a8ca1720ab7a88cfecabb8e50abf5f5a46003b998220af/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/33005b4ab8a9b179b9841acca9835831c710f88e5bed8e17e3d3283a598fbd76/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/07a2b75d3d703e315a31f51272294065ecb0872c5c881cc565760950fccdb36e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/6eca32e433b98411f9d2467b7d15b5db75b2d35f3e0fdf15fb46458501ceb466/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/4606a520a03093ffe8a7eab257409abd9087fec493e06c7e15ceaedf141e5203/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/4f2311569c941e859e3904a03e346c82365c86b189da662da3d76374ed443d9e/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/35ca1b547938161b66c6ffa6dab3af461522e5c5ce9f3ddfbdc139660390fda7/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/44d02cb6e3e3f058746c53e2f6f70749fdf3029906afb39c2166a0b68edb7761/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2d19b1e6cddbb4346a44ecef7f9e76672914ccb25417eab890ecf1c745bbcf2b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/91057a6787ab7a6d1fdfe59e5bf8dae357b421acc2e38d65b8523b19ed53e106/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cbc47cc49064572605a9d8340abba7d14c4416f870c2697001cb0ca0b47b7254/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4845b38bf439c543aa893d0f629a8ba295c1f2a90e384d45db8854424eb03c1e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c1dd844fd30bbc506631558cd30be72508e7e675c66c0d0478e1843e72265b30/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2289ade2c236bc34b6daaa79e7742b3c037e9c5969f4d26093f8e3ed37d943ee/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/9c5492ad6cd749921a6b8e155fb8eb52814f1bcea43235c7a1d357effe33e86b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/944c794ca471eda9f8f5e539c52695bc8d4e6b7ab9250b1c866221e75e5e6adb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4feee238132f89227df4aae0f40d9ce9ab355258aec75b9be6a548b4b4181ac4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ff9ebe5a19a71251fd350fa9ef3f627789572a89c4c64929c08c38041aef3fb4/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.283572 5973 manager.go:217] Machine: {Timestamp:2026-11-03 21:46:00.283106154 +0000 UTC m=+0.057955999 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099660 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2256dbfba219644f14ae359e02c655 SystemUUID:ec2256db-fba2-1964-4f14-ae359e02c655 BootID:f66d0108-2535-46e8-a947-3f5da26ebb32 Filesystems:[{Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/153a10b895cd74eebb67d1dac4c13bd0e6d1108b9a9b052cd43c8be0e1dc0ef7/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9f0f1e-3e27-4f6a-aff3-a621b458030b/volumes/kubernetes.io~projected/kube-api-access-lgwf8 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~projected/kube-api-access-sznpn DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:326 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9135178-a435-448e-b425-e821a0470160/volumes/kubernetes.io~projected/kube-api-access-7pw5x DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79c9ce6ebd8dec00c31ae2636ce943a9432611e6c03468ba7a93919574354e1f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fa6ef32ee185c95738f55eeabff484e313bcfe0da52a75e37c0f2d5b3267249/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82debff8-da8f-4dfa-980b-374e735bd8c4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~projected/kube-api-access-2f2vp DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~projected/kube-api-access-rkm9r DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de3f0fc1beba5c830dbfebb0e413615f5ac5286dbed46beb6db2875f3cde9c00/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/647406a2-8819-4286-b814-eff7c4ec3930/volumes/kubernetes.io~projected/kube-api-access-2k4dp DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6219f4bcd5c03f46075a5f338725bd10c3846acc96318385218e652b55ee424f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a5ff3a9752a967b597e1c8db6cbe20202b9d3c37ed21855b4365f6a3ee2971c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/146396b5ba9361f7504963374a7b231026ab9d91891d86d2a8110c009aceaf5d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:322 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes/kubernetes.io~projected/kube-api-access-wq5z4 DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~projected/kube-api-access-fjtfc DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49f6d106da483c2ba2ade0a4f55ab5316a1439967d76bf005a89a350470ccad6/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ff26b2d967b8b76efde4a7d8815e668b16e1347565bef7593d2d8e993188740/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc3611d44c9346787dd8bba66b83c55d914178ade1515518872c0324d49b24fd/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce72363d254044edeba5985081b479e1994c5a01eaa8832a8c6f9ce8cb8ad40f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~projected/kube-api-access-h9tx5 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-165 DeviceMajor:0 DeviceMinor:165 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cc358d16fccba56a7247b016d5adbf614018239fbfa1a74062a1e377ca1c600/userdata/shm DeviceMajor:0 DeviceMinor:368 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af58d121633f4b52c44785ca2006fea2f42b81c71f6ab0505ed119fe756a2951/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2076766-b393-4828-bf8f-30e8b64b22c4/volumes/kubernetes.io~projected/kube-api-access-qnqvm DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~projected/kube-api-access-fcgnq DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~projected/kube-api-access-fv59g DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cfb89632-d017-4912-9bd3-ae92a2e187fe/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e517e3a-a9eb-4811-99b4-1e24a4001f94/volumes/kubernetes.io~projected/kube-api-access-vb2zq DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~projected/kube-api-access-twt86 DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4374a20c-4407-4bea-a0eb-d278dd8231b1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf3b070a26a1721c5122d5bbd50d2deea3806ee3378af7d5d5369df9a1dc58b0/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/448eec2a3723864aa10bb5d6f1915a2d3dbc1f511c45fa340a60360236cc7bcc/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6619937a0fd11ddab527249c688880af53dbefb79b942b09b428b6264dcd533d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12498367-0192-4402-bb17-bfe4f3432d3a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38585226-48bc-4637-88aa-ff1cf9b9fd78/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baffe5c-d923-43b8-a3d4-7d1edf10e744/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb34c3d-2432-43b6-bfca-e172503a1bab/volumes/kubernetes.io~projected/kube-api-access-rtq9p DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3705e8-d005-4131-bc19-78c18fc31aa7/volumes/kubernetes.io~projected/kube-api-access-qwbjc DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24414154-24ae-4506-bb36-49d64ba5fcef/volumes/kubernetes.io~projected/kube-api-access-thj6p DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33267669-fbef-4d65-a6f3-23b24f74ca7f/volumes/kubernetes.io~projected/kube-api-access-pmjkb DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afddf4143b55521fa2b215ac4502806a315d51aaa37dfe5b154fc4b386654258/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78b46919-59fa-4ba1-ad2e-d226cc365941/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:366 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/volumes/kubernetes.io~projected/kube-api-access-mbtfj DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cefbfcec-8810-4b61-8cee-46091553ee7a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/feec6a26-5dc3-4688-829d-f5316d6bda7d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd5f8c1f12907ea279fbd55c0dbd1fa7680fc5a480c1a76d3baa84d477e9814/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/406bff1a87c92ee68602e1dee37c9cdfe5499db590c4f220380839401d39de67/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:146396b5ba9361f MacAddress:52:00:bf:3b:df:81 Speed:10000 Mtu:8901} {Name:153a10b895cd74e MacAddress:42:c1:56:83:08:93 Speed:10000 Mtu:8901} {Name:1ff26b2d967b8b7 MacAddress:56:07:d5:c5:bd:41 Speed:10000 Mtu:8901} {Name:2dd5f8c1f12907e MacAddress:a2:db:cc:83:de:ac Speed:10000 Mtu:8901} {Name:37042f6cb1a3e5a MacAddress:4e:5b:1f:d3:b4:05 Speed:10000 Mtu:8901} {Name:406bff1a87c92ee MacAddress:ee:4a:e1:1e:31:dc Speed:10000 Mtu:8901} {Name:7cc358d16fccba5 MacAddress:7e:28:09:7c:cd:fd Speed:10000 Mtu:8901} {Name:af58d121633f4b5 MacAddress:46:53:f6:71:ee:15 Speed:10000 Mtu:8901} {Name:afddf4143b55521 MacAddress:8a:fa:fe:0f:65:84 Speed:10000 Mtu:8901} {Name:bc3611d44c93467 MacAddress:22:99:e2:5a:58:3b Speed:10000 Mtu:8901} {Name:bf3b070a26a1721 MacAddress:e2:de:80:b3:51:85 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:4c:38:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:23:36:62:6c:0d Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:4c:38:11 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ca:4b:60:2b:5a:23 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:70:9d:d1:23:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.283912 5973 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.283951 5973 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284223 5973 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284244 5973 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-89.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284371 5973 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284380 5973 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284392 5973 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284408 5973 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284488 5973 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284654 5973 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284749 5973 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284780 5973 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284789 5973 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284797 5973 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284806 5973 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:46:00.285258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.284815 5973 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:46:00.286360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.285374 5973 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:46:00.286360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.285383 5973 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:46:00.286360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.285838 5973 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:46:00.288495 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288479 5973 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:46:00.288495 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288493 5973 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:46:00.288618 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288585 5973 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 21:46:00.288776 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288765 5973 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:46:00.288810 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288794 5973 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:46:00.288885 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288876 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:46:00.288925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288890 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:46:00.288925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288897 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:46:00.288925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288903 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:46:00.288925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288917 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288927 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288935 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288941 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288947 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288953 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288960 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288971 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.288998 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:46:00.289029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.289007 5973 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:46:00.292581 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.292567 5973 server.go:1263] "Started kubelet" Nov 03 21:46:00.293382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.292640 5973 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:46:00.293382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.292659 5973 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:46:00.293382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.292731 5973 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:46:00.293382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.293166 5973 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:46:00.293115 ip-10-0-131-89 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:46:00.296343 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.295516 5973 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:46:00.298511 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298495 5973 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298601 5973 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.298654 5973 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-89.us-east-2.compute.internal\" not found" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298644 5973 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:39:51 +0000 UTC" deadline="2028-06-19 15:03:15.000991794 +0000 UTC" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298688 5973 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298691 5973 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14249h17m14.702306698s" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298697 5973 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.298710 5973 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.300423 5973 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.300440 5973 factory.go:55] Registering systemd factory Nov 03 21:46:00.301152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.300450 5973 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301527 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a9e1cd-c054-453d-bd1f-fdaa0a35e700" volumeName="kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301562 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9135178-a435-448e-b425-e821a0470160" volumeName="kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301577 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9135178-a435-448e-b425-e821a0470160" volumeName="kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301588 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301600 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301612 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38585226-48bc-4637-88aa-ff1cf9b9fd78" volumeName="kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301623 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e517e3a-a9eb-4811-99b4-1e24a4001f94" volumeName="kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301635 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 21:46:00.301647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301650 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="647406a2-8819-4286-b814-eff7c4ec3930" volumeName="kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301663 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78b46919-59fa-4ba1-ad2e-d226cc365941" volumeName="kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301676 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb34c3d-2432-43b6-bfca-e172503a1bab" volumeName="kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301614 5973 factory.go:153] Registering CRI-O factory Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301705 5973 factory.go:223] Registration of the crio container factory successfully Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301727 5973 factory.go:103] Registering Raw factory Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301742 5973 manager.go:1199] Started watching for new ooms in manager Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301687 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301781 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301801 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85" volumeName="kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301829 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301841 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a9e1cd-c054-453d-bd1f-fdaa0a35e700" volumeName="kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301852 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301860 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301867 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301897 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78b46919-59fa-4ba1-ad2e-d226cc365941" volumeName="kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301906 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301925 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feec6a26-5dc3-4688-829d-f5316d6bda7d" volumeName="kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301935 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12498367-0192-4402-bb17-bfe4f3432d3a" volumeName="kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301943 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301953 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9f0f1e-3e27-4f6a-aff3-a621b458030b" volumeName="kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301961 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="647406a2-8819-4286-b814-eff7c4ec3930" volumeName="kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301969 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85" volumeName="kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301975 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301987 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3705e8-d005-4131-bc19-78c18fc31aa7" volumeName="kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.301995 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2076766-b393-4828-bf8f-30e8b64b22c4" volumeName="kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302018 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38585226-48bc-4637-88aa-ff1cf9b9fd78" volumeName="kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302030 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38585226-48bc-4637-88aa-ff1cf9b9fd78" volumeName="kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302038 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38585226-48bc-4637-88aa-ff1cf9b9fd78" volumeName="kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302047 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302055 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302064 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302076 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 21:46:00.302069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302083 5973 manager.go:319] Starting recovery of all containers Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.302084 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a9e1cd-c054-453d-bd1f-fdaa0a35e700" volumeName="kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303703 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85" volumeName="kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303734 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303750 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303762 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303779 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2076766-b393-4828-bf8f-30e8b64b22c4" volumeName="kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303792 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2076766-b393-4828-bf8f-30e8b64b22c4" volumeName="kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303810 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feec6a26-5dc3-4688-829d-f5316d6bda7d" volumeName="kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303830 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9f0f1e-3e27-4f6a-aff3-a621b458030b" volumeName="kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303845 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12498367-0192-4402-bb17-bfe4f3432d3a" volumeName="kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303867 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303886 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303904 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303918 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303930 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2076766-b393-4828-bf8f-30e8b64b22c4" volumeName="kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303948 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303959 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85" volumeName="kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303978 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfb89632-d017-4912-9bd3-ae92a2e187fe" volumeName="kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.303990 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4374a20c-4407-4bea-a0eb-d278dd8231b1" volumeName="kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304024 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304040 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82debff8-da8f-4dfa-980b-374e735bd8c4" volumeName="kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304053 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefbfcec-8810-4b61-8cee-46091553ee7a" volumeName="kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304070 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304081 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24414154-24ae-4506-bb36-49d64ba5fcef" volumeName="kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304094 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304111 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304123 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304142 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="647406a2-8819-4286-b814-eff7c4ec3930" volumeName="kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304162 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefbfcec-8810-4b61-8cee-46091553ee7a" volumeName="kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304174 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4374a20c-4407-4bea-a0eb-d278dd8231b1" volumeName="kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304190 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb34c3d-2432-43b6-bfca-e172503a1bab" volumeName="kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304203 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefbfcec-8810-4b61-8cee-46091553ee7a" volumeName="kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304220 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a9e1cd-c054-453d-bd1f-fdaa0a35e700" volumeName="kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304234 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefbfcec-8810-4b61-8cee-46091553ee7a" volumeName="kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304248 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefbfcec-8810-4b61-8cee-46091553ee7a" volumeName="kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304264 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33267669-fbef-4d65-a6f3-23b24f74ca7f" volumeName="kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304277 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4374a20c-4407-4bea-a0eb-d278dd8231b1" volumeName="kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304294 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baffe5c-d923-43b8-a3d4-7d1edf10e744" volumeName="kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304307 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304320 5973 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a74320bc-ea6d-4469-98e7-7440ba94cef2" volumeName="kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4" seLinuxMountContext="" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304336 5973 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:46:00.304586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.304343 5973 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:46:00.307495 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.307446 5973 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:46:00.308036 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.308018 5973 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:46:00.311899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.311860 5973 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 21:46:00.311899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.311864 5973 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:46:00.336569 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.336538 5973 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:46:00.338067 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.338049 5973 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:46:00.338067 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.338066 5973 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:46:00.338175 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.338089 5973 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:46:00.338175 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.338121 5973 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 21:46:00.342082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.342059 5973 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:46:00.363200 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.363176 5973 generic.go:356] "Generic (PLEG): container finished" podID="33267669-fbef-4d65-a6f3-23b24f74ca7f" containerID="80344c85b74671ddb3fcc4f050eacbab8f1794eb31ff9de58e46649303c36c14" exitCode=0 Nov 03 21:46:00.373730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373707 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="889b3bb5431bd17f336bd9ac5ea055f0d60e658dd988e014de519c74b5800f07" exitCode=0 Nov 03 21:46:00.373730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373726 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="0eb18176a128b0b3ed391a584a8697b5ad2d2c4371a8965269fdafaefd39041c" exitCode=0 Nov 03 21:46:00.373730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373732 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="73e5bd4819d818f64ee06161f3834dbf3270b13a73884571783e01d7f65d543a" exitCode=0 Nov 03 21:46:00.373730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373737 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="0ee0904e2224a2ecb8c4faa3fdd56aea474e4a88dbfcbfb4f7ec2fc0927c2e11" exitCode=0 Nov 03 21:46:00.373986 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373742 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="aa49f16a7252536c8dbae478d0ebfec497c98c17c0383d035c6098cb20ef1559" exitCode=0 Nov 03 21:46:00.373986 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.373747 5973 generic.go:356] "Generic (PLEG): container finished" podID="f2076766-b393-4828-bf8f-30e8b64b22c4" containerID="abf4c02918c4c412d2ca24381e9e148b4e31809742f90081afa7e1d0cc05c10c" exitCode=0 Nov 03 21:46:00.381908 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.381882 5973 generic.go:356] "Generic (PLEG): container finished" podID="5d7d2a1c76cdd1f6aba5eb1c53d3f6b0" containerID="fc56af8042e0883965349e4c2769a9a24cba31d063efd20ca87ced3a2350f536" exitCode=0 Nov 03 21:46:00.405325 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.405290 5973 manager.go:324] Recovery completed Nov 03 21:46:00.425746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425717 5973 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:46:00.425746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425737 5973 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:46:00.425888 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425758 5973 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:46:00.425888 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425880 5973 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425891 5973 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425907 5973 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425913 5973 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425918 5973 policy_none.go:50] "Start" Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425926 5973 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:46:00.425974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.425934 5973 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:46:00.426146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.426052 5973 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 21:46:00.426146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.426059 5973 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 21:46:00.426146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.426064 5973 policy_none.go:44] "Start" Nov 03 21:46:00.430526 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.430450 5973 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:46:00.435116 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.435104 5973 manager.go:341] "Starting Device Plugin manager" Nov 03 21:46:00.435794 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.435127 5973 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:46:00.435832 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.435800 5973 server.go:87] "Starting device plugin registration server" Nov 03 21:46:00.436100 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436088 5973 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:46:00.436150 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436102 5973 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:46:00.436225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436208 5973 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:46:00.436351 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436312 5973 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:46:00.436451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436357 5973 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:46:00.436451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436364 5973 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:46:00.436577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.436410 5973 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:46:00.43634611Z","UUID":"0ce3d0d5-8ef5-413e-a5d6-fdf03a5959c2","Handler":null,"Name":"","Endpoint":""} Nov 03 21:46:00.438002 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.437977 5973 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:46:00.438329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.438291 5973 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal"] Nov 03 21:46:00.438426 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.438406 5973 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:46:00.438519 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.438504 5973 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:46:00.439889 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.439846 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"a387975d67fca887e2f3c912beb1760d","Type":"ContainerStarted","Data":"7fcdc75cdef329d15164e64abe2545777fa57c9796fe8033e5a729f56b6fddc4"} Nov 03 21:46:00.439975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.439907 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"a387975d67fca887e2f3c912beb1760d","Type":"ContainerStarted","Data":"db2c74aecc077c4c570d9b981c36c31d776b20e3d7f16d3f9af7c213df3ce875"} Nov 03 21:46:00.439975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.439943 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerStarted","Data":"aa002c79a54e1994bb292b1038aa8b81e30756d5d0db4aecc7b2d8db4e6a550e"} Nov 03 21:46:00.439975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.439967 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerDied","Data":"fc56af8042e0883965349e4c2769a9a24cba31d063efd20ca87ced3a2350f536"} Nov 03 21:46:00.440774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.439988 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" event={"ID":"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0","Type":"ContainerStarted","Data":"1bb7ed009e3f34a5c81851e8b4c6fa6749cad11839f8b7a77aa0beba86786b59"} Nov 03 21:46:00.440774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.440374 5973 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.440774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.440424 5973 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.450681 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.450665 5973 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:46:00.450779 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.450713 5973 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.451390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.451372 5973 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:46:00.451497 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:00.451420 5973 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.512784 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.512747 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.512870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.512791 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.512870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.512815 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.536872 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.536852 5973 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:46:00.538782 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.538763 5973 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 21:46:00.538847 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.538792 5973 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 21:46:00.538847 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.538802 5973 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-89.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 21:46:00.538847 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.538840 5973 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.551327 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.551307 5973 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.551379 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.551356 5973 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.612937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.612913 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.612937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.612928 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.613082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.612957 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.613082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.612980 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.613082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.613016 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a387975d67fca887e2f3c912beb1760d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"a387975d67fca887e2f3c912beb1760d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:00.613082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:00.613021 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7d2a1c76cdd1f6aba5eb1c53d3f6b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal\" (UID: \"5d7d2a1c76cdd1f6aba5eb1c53d3f6b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal" Nov 03 21:46:01.285991 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.285959 5973 apiserver.go:52] "Watching apiserver" Nov 03 21:46:01.298229 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.298205 5973 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:46:01.298918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.298890 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-t265q","openshift-monitoring/node-exporter-fnlbr","openshift-multus/multus-hvj6c","openshift-multus/network-metrics-daemon-f9v7m","openshift-network-operator/iptables-alerter-thpfp","openshift-ovn-kubernetes/ovnkube-node-rxr85","openshift-dns/node-resolver-bnwgp","openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl","openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh","kube-system/global-pull-secret-syncer-cw9gc","openshift-monitoring/telemeter-client-75f954c659-h7nqb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb","openshift-console/console-5c56dbb448-ftbq5","openshift-insights/insights-runtime-extractor-snnch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.us-east-2.compute.internal","openshift-dns/dns-default-hvnmj","openshift-ingress-canary/ingress-canary-8j8zm","openshift-multus/multus-additional-cni-plugins-6plmd","openshift-cluster-node-tuning-operator/tuned-kdggf","kube-system/konnectivity-agent-hx8cd","openshift-monitoring/metrics-server-79b56c756-lg8dh","openshift-network-diagnostics/network-check-target-f6h64","kube-system/kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal"] Nov 03 21:46:01.303840 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.303814 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lbb62\"" Nov 03 21:46:01.303840 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.303814 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jbpmc\"" Nov 03 21:46:01.304071 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304057 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.304179 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304161 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.304258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304245 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:46:01.304307 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304261 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.304893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304874 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:46:01.304893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304874 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.305017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.304968 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:46:01.305084 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.305062 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rdd92\"" Nov 03 21:46:01.305148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.305126 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:46:01.311843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.311823 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.312080 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312060 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:46:01.312080 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312062 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.312249 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312231 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.312300 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312271 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:46:01.312333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312306 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.312880 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.312865 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:46:01.313523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.313505 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.313597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.313508 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.313597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.313530 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:46:01.313597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.313513 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:46:01.313597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.313565 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.314960 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.314946 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p72km\"" Nov 03 21:46:01.315017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.314967 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.315017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.314968 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.315072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315017 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.315450 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315431 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.315557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315491 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.315557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315504 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t98zr\"" Nov 03 21:46:01.315557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315516 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:46:01.315557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315530 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hgwjm\"" Nov 03 21:46:01.315557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.315557 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.316157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316130 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:46:01.316225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316171 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:46:01.316225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316176 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7bvj7\"" Nov 03 21:46:01.316350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316232 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:46:01.316350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316240 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316385 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316395 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fc4bx\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316415 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316427 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316419 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5xcnq\"" Nov 03 21:46:01.316447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316440 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lqk2r\"" Nov 03 21:46:01.316688 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316429 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-txfgg\"" Nov 03 21:46:01.316688 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316504 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bc4mv\"" Nov 03 21:46:01.316953 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316937 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:46:01.317008 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316957 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52n7c\"" Nov 03 21:46:01.317008 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.316978 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:46:01.317082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317017 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wc5kn\"" Nov 03 21:46:01.317082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317032 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.318117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317821 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:46:01.318117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317853 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 21:46:01.318117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317885 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.318117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317942 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 21:46:01.318117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.317999 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:46:01.318393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318132 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 21:46:01.318393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318278 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:46:01.318393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318284 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-92tpk\"" Nov 03 21:46:01.318393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318317 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:46:01.318848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318826 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:46:01.318848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318855 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 21:46:01.319000 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318935 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 21:46:01.319000 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.318980 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 21:46:01.319527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319506 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 21:46:01.319602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319556 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 21:46:01.319602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319588 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vr8v6\"" Nov 03 21:46:01.319704 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319648 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9smcc\"" Nov 03 21:46:01.319752 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319723 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 21:46:01.319861 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319842 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 21:46:01.320004 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.319970 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 21:46:01.320234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.320210 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5f4x\"" Nov 03 21:46:01.320356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.320335 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 21:46:01.320512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.320498 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 21:46:01.320872 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.320852 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 21:46:01.320964 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.320894 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nqlh5\"" Nov 03 21:46:01.321095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.321077 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2g3ek12ouge4r\"" Nov 03 21:46:01.321147 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.321127 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 21:46:01.324443 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.324425 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 21:46:01.400981 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.400954 5973 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:46:01.403992 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.403965 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 21:46:01.404117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.404018 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wc92d\"" Nov 03 21:46:01.404117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.404035 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 21:46:01.416386 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416360 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.416490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416391 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.416490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416412 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.416490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416436 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.416592 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416564 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.416622 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416594 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thj6p\" (UniqueName: \"kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.416662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416625 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.416662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416635 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb34c3d-2432-43b6-bfca-e172503a1bab-iptables-alerter-script\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.416662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416653 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416678 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416688 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-script-lib\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.416767 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416812 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-env-overrides\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416819 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.416848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416814 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.416855 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.916836453 +0000 UTC m=+1.691686309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416878 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416905 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416921 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416937 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.416936 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-tls\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417076 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417102 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417121 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgnq\" (UniqueName: \"kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:46:01.417142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417137 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417188 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417210 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznpn\" (UniqueName: \"kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417212 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-tls\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417227 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417242 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtfj\" (UniqueName: \"kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417257 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417276 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt86\" (UniqueName: \"kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417298 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417352 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38585226-48bc-4637-88aa-ff1cf9b9fd78-metrics-tls\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417414 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.417451 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417485 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417511 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4dp\" (UniqueName: \"kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417512 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-serving-certs-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417531 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.417543 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.91752975 +0000 UTC m=+1.692379602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417588 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:46:01.417627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417641 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417658 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417673 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417785 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-serviceca\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417805 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12498367-0192-4402-bb17-bfe4f3432d3a-cert\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417819 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417852 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cefbfcec-8810-4b61-8cee-46091553ee7a-ovnkube-config\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417855 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417863 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417879 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tx5\" (UniqueName: \"kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417909 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417935 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417958 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.417981 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418005 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418045 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418115 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418124 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-textfile\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418195 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418217 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feec6a26-5dc3-4688-829d-f5316d6bda7d-agent-certs\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418238 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2vp\" (UniqueName: \"kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.418250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418259 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418280 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418301 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418318 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418335 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418297 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-metrics-server-audit-profiles\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418368 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418379 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418407 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418488 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418498 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9135178-a435-448e-b425-e821a0470160-tmp-dir\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418490 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418537 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418568 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418603 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-data-volume\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418610 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418628 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418652 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418668 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418708 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.418742 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418763 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418779 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418829 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418857 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.418870 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.918859746 +0000 UTC m=+1.693709571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418922 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418968 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418984 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.418988 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419004 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-secret-telemeter-client\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419007 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419062 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-secret-metrics-server-client-certs\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419067 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwf8\" (UniqueName: \"kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419110 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feec6a26-5dc3-4688-829d-f5316d6bda7d-konnectivity-ca\") pod \"konnectivity-agent-hx8cd\" (UID: \"feec6a26-5dc3-4688-829d-f5316d6bda7d\") " pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419119 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419149 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419174 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.419186 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419194 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419307 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnqvm\" (UniqueName: \"kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419342 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419359 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419362 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419359 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419403 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419490 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419539 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419579 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419648 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419672 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjkb\" (UniqueName: \"kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419726 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw5x\" (UniqueName: \"kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419790 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419820 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419848 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419871 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419911 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419969 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.419970 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb89632-d017-4912-9bd3-ae92a2e187fe-audit-log\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420027 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-tuned\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420046 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420063 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420082 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420097 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420113 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420136 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv59g\" (UniqueName: \"kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420162 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm9r\" (UniqueName: \"kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420154 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb89632-d017-4912-9bd3-ae92a2e187fe-client-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420196 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420224 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33267669-fbef-4d65-a6f3-23b24f74ca7f-metrics-client-ca\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.420230 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420294 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.420317 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.920307362 +0000 UTC m=+1.695157187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420355 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420387 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420406 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb89632-d017-4912-9bd3-ae92a2e187fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420413 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420428 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82debff8-da8f-4dfa-980b-374e735bd8c4-original-pull-secret\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420439 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420509 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420541 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420561 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420622 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420650 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420653 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420667 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420670 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5baffe5c-d923-43b8-a3d4-7d1edf10e744-federate-client-tls\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420702 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420724 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420742 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420762 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4374a20c-4407-4bea-a0eb-d278dd8231b1-tmp\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.420775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420808 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420836 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420904 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-metrics-client-ca\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420931 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b46919-59fa-4ba1-ad2e-d226cc365941-metrics-certs\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420944 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-multus-daemon-config\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420940 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420974 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq9p\" (UniqueName: \"kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.420996 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421012 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421022 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38585226-48bc-4637-88aa-ff1cf9b9fd78-config-volume\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421028 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421062 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421099 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421108 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/24414154-24ae-4506-bb36-49d64ba5fcef-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421118 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2076766-b393-4828-bf8f-30e8b64b22c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421123 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421192 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38585226-48bc-4637-88aa-ff1cf9b9fd78-tmp-dir\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421196 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24414154-24ae-4506-bb36-49d64ba5fcef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421218 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbjc\" (UniqueName: \"kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421248 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421285 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421404 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421430 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421487 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421505 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421522 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.421555 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.421603 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.921589286 +0000 UTC m=+1.696439129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421605 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cefbfcec-8810-4b61-8cee-46091553ee7a-ovn-node-metrics-cert\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421630 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.421664 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421673 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421687 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647406a2-8819-4286-b814-eff7c4ec3930-cni-binary-copy\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421696 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.421707 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.921694029 +0000 UTC m=+1.696543854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421838 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421862 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421908 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtfc\" (UniqueName: \"kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421925 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5baffe5c-d923-43b8-a3d4-7d1edf10e744-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421951 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.421992 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.422021 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.422022 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.422057 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.422272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.422085 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.430521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.430500 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thj6p\" (UniqueName: \"kubernetes.io/projected/24414154-24ae-4506-bb36-49d64ba5fcef-kube-api-access-thj6p\") pod \"kube-state-metrics-5dc8d7d89b-h8hsl\" (UID: \"24414154-24ae-4506-bb36-49d64ba5fcef\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h8hsl" Nov 03 21:46:01.431269 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.431253 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2vp\" (UniqueName: \"kubernetes.io/projected/aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85-kube-api-access-2f2vp\") pod \"openshift-state-metrics-7bd7489957-lg4xh\" (UID: \"aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lg4xh" Nov 03 21:46:01.433481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.433448 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgnq\" (UniqueName: \"kubernetes.io/projected/78b46919-59fa-4ba1-ad2e-d226cc365941-kube-api-access-fcgnq\") pod \"network-metrics-daemon-f9v7m\" (UID: \"78b46919-59fa-4ba1-ad2e-d226cc365941\") " pod="openshift-multus/network-metrics-daemon-f9v7m" Nov 03 21:46:01.435748 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.435731 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:01.435788 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.435748 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:01.435788 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.435758 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:01.435888 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.435811 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:01.935800292 +0000 UTC m=+1.710650118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:01.436166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.436151 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tx5\" (UniqueName: \"kubernetes.io/projected/4374a20c-4407-4bea-a0eb-d278dd8231b1-kube-api-access-h9tx5\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.436294 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.436277 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4dp\" (UniqueName: \"kubernetes.io/projected/647406a2-8819-4286-b814-eff7c4ec3930-kube-api-access-2k4dp\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.438899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.438872 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw5x\" (UniqueName: \"kubernetes.io/projected/a9135178-a435-448e-b425-e821a0470160-kube-api-access-7pw5x\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.439202 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439175 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznpn\" (UniqueName: \"kubernetes.io/projected/5baffe5c-d923-43b8-a3d4-7d1edf10e744-kube-api-access-sznpn\") pod \"telemeter-client-75f954c659-h7nqb\" (UID: \"5baffe5c-d923-43b8-a3d4-7d1edf10e744\") " pod="openshift-monitoring/telemeter-client-75f954c659-h7nqb" Nov 03 21:46:01.439282 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439223 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt86\" (UniqueName: \"kubernetes.io/projected/38585226-48bc-4637-88aa-ff1cf9b9fd78-kube-api-access-twt86\") pod \"dns-default-hvnmj\" (UID: \"38585226-48bc-4637-88aa-ff1cf9b9fd78\") " pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:01.439339 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439313 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv59g\" (UniqueName: \"kubernetes.io/projected/12498367-0192-4402-bb17-bfe4f3432d3a-kube-api-access-fv59g\") pod \"ingress-canary-8j8zm\" (UID: \"12498367-0192-4402-bb17-bfe4f3432d3a\") " pod="openshift-ingress-canary/ingress-canary-8j8zm" Nov 03 21:46:01.439339 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439316 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwf8\" (UniqueName: \"kubernetes.io/projected/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-kube-api-access-lgwf8\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.439485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.438527 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq9p\" (UniqueName: \"kubernetes.io/projected/7cb34c3d-2432-43b6-bfca-e172503a1bab-kube-api-access-rtq9p\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.439485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439415 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2zq\" (UniqueName: \"kubernetes.io/projected/8e517e3a-a9eb-4811-99b4-1e24a4001f94-kube-api-access-vb2zq\") pod \"network-check-target-f6h64\" (UID: \"8e517e3a-a9eb-4811-99b4-1e24a4001f94\") " pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:46:01.439627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439552 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm9r\" (UniqueName: \"kubernetes.io/projected/cfb89632-d017-4912-9bd3-ae92a2e187fe-kube-api-access-rkm9r\") pod \"metrics-server-79b56c756-lg8dh\" (UID: \"cfb89632-d017-4912-9bd3-ae92a2e187fe\") " pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:01.439708 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.439686 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtfj\" (UniqueName: \"kubernetes.io/projected/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-kube-api-access-mbtfj\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.440211 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.440189 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtfc\" (UniqueName: \"kubernetes.io/projected/cefbfcec-8810-4b61-8cee-46091553ee7a-kube-api-access-fjtfc\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.441130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.441105 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjkb\" (UniqueName: \"kubernetes.io/projected/33267669-fbef-4d65-a6f3-23b24f74ca7f-kube-api-access-pmjkb\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.441401 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.441379 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbjc\" (UniqueName: \"kubernetes.io/projected/da3705e8-d005-4131-bc19-78c18fc31aa7-kube-api-access-qwbjc\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.441636 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.441615 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnqvm\" (UniqueName: \"kubernetes.io/projected/f2076766-b393-4828-bf8f-30e8b64b22c4-kube-api-access-qnqvm\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.523231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523190 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523236 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523256 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523278 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523302 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523323 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-system-cni-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523341 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-log-socket\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523355 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-socket-dir-parent\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523384 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a9e1cd-c054-453d-bd1f-fdaa0a35e700-crio-socket\") pod \"insights-runtime-extractor-snnch\" (UID: \"d5a9e1cd-c054-453d-bd1f-fdaa0a35e700\") " pod="openshift-insights/insights-runtime-extractor-snnch" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523388 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523414 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523442 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb34c3d-2432-43b6-bfca-e172503a1bab-host-slash\") pod \"iptables-alerter-thpfp\" (UID: \"7cb34c3d-2432-43b6-bfca-e172503a1bab\") " pod="openshift-network-operator/iptables-alerter-thpfp" Nov 03 21:46:01.523483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523447 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523501 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523508 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-device-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523517 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523535 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523535 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-netns\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523558 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-kubelet\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523562 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523603 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523625 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523634 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-systemd\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523640 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523644 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysctl-conf\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523655 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523658 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-ovn\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523670 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-systemd-units\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523672 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523689 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523710 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523725 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-sys\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523767 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523813 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523847 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523845 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523875 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523893 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-k8s-cni-cncf-io\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523895 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523913 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-root\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523893 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-kubelet-config\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523923 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.523939 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523938 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523964 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523982 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523984 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524001 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-lib-modules\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524014 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524031 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524041 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.523966 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-bin\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524005 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-var-lib-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524079 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9135178-a435-448e-b425-e821a0470160-hosts-file\") pod \"node-resolver-bnwgp\" (UID: \"a9135178-a435-448e-b425-e821a0470160\") " pod="openshift-dns/node-resolver-bnwgp" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524066 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524095 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-hostroot\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524106 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33267669-fbef-4d65-a6f3-23b24f74ca7f-node-exporter-wtmp\") pod \"node-exporter-fnlbr\" (UID: \"33267669-fbef-4d65-a6f3-23b24f74ca7f\") " pod="openshift-monitoring/node-exporter-fnlbr" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524114 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524138 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524167 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524181 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-sys\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524197 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524205 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-cni-netd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524214 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-host\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524244 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524260 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524279 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524305 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524330 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524335 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c9f0f1e-3e27-4f6a-aff3-a621b458030b-host\") pod \"node-ca-t265q\" (UID: \"0c9f0f1e-3e27-4f6a-aff3-a621b458030b\") " pod="openshift-image-registry/node-ca-t265q" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524307 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-slash\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524358 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524374 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524378 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-sys-fs\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524388 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524387 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524404 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-run-multus-certs\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524419 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2076766-b393-4828-bf8f-30e8b64b22c4-cnibin\") pod \"multus-additional-cni-plugins-6plmd\" (UID: \"f2076766-b393-4828-bf8f-30e8b64b22c4\") " pod="openshift-multus/multus-additional-cni-plugins-6plmd" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524433 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-registration-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524451 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524493 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-multus\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524496 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524526 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-etc-openvswitch\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524529 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524544 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-host-var-lib-cni-bin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524564 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524589 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524608 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-kubelet\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524627 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524652 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524673 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-run-systemd\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524674 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524698 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524711 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-system-cni-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524702 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-kubernetes\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524721 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-node-log\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524725 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524750 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82debff8-da8f-4dfa-980b-374e735bd8c4-dbus\") pod \"global-pull-secret-syncer-cw9gc\" (UID: \"82debff8-da8f-4dfa-980b-374e735bd8c4\") " pod="kube-system/global-pull-secret-syncer-cw9gc" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524755 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524778 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524793 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-modprobe-d\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524794 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524811 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-var-lib-kubelet\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524815 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524820 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-cnibin\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524834 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-etc-kubernetes\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524841 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524843 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647406a2-8819-4286-b814-eff7c4ec3930-multus-conf-dir\") pod \"multus-hvj6c\" (UID: \"647406a2-8819-4286-b814-eff7c4ec3930\") " pod="openshift-multus/multus-hvj6c" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524870 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da3705e8-d005-4131-bc19-78c18fc31aa7-socket-dir\") pod \"aws-ebs-csi-driver-node-7gggb\" (UID: \"da3705e8-d005-4131-bc19-78c18fc31aa7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gggb" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524877 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524900 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524919 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-etc-sysconfig\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524926 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.524941 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cefbfcec-8810-4b61-8cee-46091553ee7a-host-run-netns\") pod \"ovnkube-node-rxr85\" (UID: \"cefbfcec-8810-4b61-8cee-46091553ee7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:01.525157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.525009 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4374a20c-4407-4bea-a0eb-d278dd8231b1-run\") pod \"tuned-kdggf\" (UID: \"4374a20c-4407-4bea-a0eb-d278dd8231b1\") " pod="openshift-cluster-node-tuning-operator/tuned-kdggf" Nov 03 21:46:01.928182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928145 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928189 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928221 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928279 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928286 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928301 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928328 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928360 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928347593 +0000 UTC m=+2.703197418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928386 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928390 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:01.928421 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928420 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928405018 +0000 UTC m=+2.703254850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928390 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928443 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928431583 +0000 UTC m=+2.703281419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928493 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928484916 +0000 UTC m=+2.703334745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928541 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928531919 +0000 UTC m=+2.703381750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:01.928592 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928664 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:01.928857 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:01.928722 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:02.928711099 +0000 UTC m=+2.703560937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:02.029565 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.029532 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.029701 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.029668 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:02.029701 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.029681 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:02.029701 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.029692 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:02.029826 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.029736 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:03.029724463 +0000 UTC m=+2.804574302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:02.936191 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936141 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936211 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936234 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936254 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936265 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936300 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936327 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936308247 +0000 UTC m=+4.711158071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936344 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936355 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936358 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936372 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936388 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936377061 +0000 UTC m=+4.711226886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936403 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936394983 +0000 UTC m=+4.711244813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:02.936380 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936418 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936424 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936409283 +0000 UTC m=+4.711259128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936441 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936434866 +0000 UTC m=+4.711284690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:02.936640 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:02.936453 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:04.936447124 +0000 UTC m=+4.711296949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:03.037168 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:03.037143 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:03.037283 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:03.037274 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:03.037340 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:03.037291 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:03.037340 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:03.037300 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:03.037340 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:03.037338 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:05.037328067 +0000 UTC m=+4.812177925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:03.576251 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:03.576220 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hvnmj" Nov 03 21:46:04.435605 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.435576 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:04.955353 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955319 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955366 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955387 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955408 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955432 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955469 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:04.955483 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955485 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955506 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955506 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955549 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955550 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.955530859 +0000 UTC m=+8.730380689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:04.955580 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955578 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:04.956068 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955596 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.955584831 +0000 UTC m=+8.730434656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:04.956068 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955611 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.95560294 +0000 UTC m=+8.730452766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:04.956068 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955622 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.955617214 +0000 UTC m=+8.730467038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:04.956068 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955632 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.955626715 +0000 UTC m=+8.730476541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:04.956068 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:04.955647 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:08.955641495 +0000 UTC m=+8.730491320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:05.056471 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:05.056427 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:05.056618 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:05.056567 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:05.056618 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:05.056587 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:05.056618 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:05.056597 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:05.056736 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:05.056640 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:09.05662703 +0000 UTC m=+8.831476874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:05.856941 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:05.856909 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:06.004237 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:06.004203 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:06.140606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:06.140544 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:46:07.171260 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:07.171236 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:07.171656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:07.171436 5973 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:46:08.996694 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.996654 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.996704 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996772 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996792 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996836 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.996823099 +0000 UTC m=+16.771672962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.996837 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996850 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.996843985 +0000 UTC m=+16.771693810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.996902 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996923 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.996941 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996964 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.996949009 +0000 UTC m=+16.771798834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.996991 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.997026 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:08.997029 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.997037 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.997024348 +0000 UTC m=+16.771874175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.997063 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.997049972 +0000 UTC m=+16.771899815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.997072 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:08.997201 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:08.997120 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:16.997111213 +0000 UTC m=+16.771961039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:09.097987 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:09.097953 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:09.098125 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:09.098109 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:09.098168 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:09.098129 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:09.098168 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:09.098140 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:09.098246 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:09.098183 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:17.098170704 +0000 UTC m=+16.873020529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:10.249726 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:10.249651 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxr85" Nov 03 21:46:12.894543 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:12.894515 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79b56c756-lg8dh" Nov 03 21:46:13.926242 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:13.926208 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:14.076263 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:14.076231 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:14.397528 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:14.397487 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:46:14.434489 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:14.434439 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:14.583093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:14.583060 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:17.062008 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.061969 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.062020 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.062041 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.062076 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062103 5973 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062145 5973 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062146 5973 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.062108 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062189 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062168177 +0000 UTC m=+32.837018031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"service-ca" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062189 5973 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062210 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062202938 +0000 UTC m=+32.837052762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-config" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062222 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062216746 +0000 UTC m=+32.837066571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-oauth-config" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062246 5973 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062254 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062235793 +0000 UTC m=+32.837085621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.062359 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062414 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062399111 +0000 UTC m=+32.837248941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"console-serving-cert" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062429 5973 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:17.062490 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.062502 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.062488726 +0000 UTC m=+32.837338554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 21:46:17.162691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:17.162668 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"console-5c56dbb448-ftbq5\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:17.162828 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.162810 5973 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 21:46:17.162863 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.162832 5973 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 21:46:17.162863 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.162842 5973 projected.go:196] Error preparing data for projected volume kube-api-access-wq5z4 for pod openshift-console/console-5c56dbb448-ftbq5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:17.162922 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:17.162883 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4 podName:a74320bc-ea6d-4469-98e7-7440ba94cef2 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:33.162870035 +0000 UTC m=+32.937719877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq5z4" (UniqueName: "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4") pod "console-5c56dbb448-ftbq5" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 21:46:18.162517 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.162485 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-699c6fd85b-jjs28"] Nov 03 21:46:18.167346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.167323 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.174793 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.174770 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nmvdk\"" Nov 03 21:46:18.174917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.174855 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 21:46:18.174917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.174863 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 21:46:18.178467 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.178437 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 21:46:18.178756 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.178741 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 21:46:18.178806 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.178763 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 21:46:18.178855 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.178843 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 21:46:18.179012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.178991 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 21:46:18.183070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.183048 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 21:46:18.183466 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.183433 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699c6fd85b-jjs28"] Nov 03 21:46:18.272363 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272325 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272380 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfds\" (UniqueName: \"kubernetes.io/projected/1b294cd0-8239-4ec3-ad32-adff86b0f779-kube-api-access-lmfds\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272479 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-oauth-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272520 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-trusted-ca-bundle\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272637 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272541 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-oauth-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272637 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272606 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272637 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272630 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b294cd0-8239-4ec3-ad32-adff86b0f779-tmp\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.272727 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.272681 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-service-ca\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.304832 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.304803 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.373952 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.373928 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-oauth-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.373955 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.373971 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b294cd0-8239-4ec3-ad32-adff86b0f779-tmp\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374009 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-service-ca\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374116 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374153 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfds\" (UniqueName: \"kubernetes.io/projected/1b294cd0-8239-4ec3-ad32-adff86b0f779-kube-api-access-lmfds\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374186 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-oauth-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374309 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374289 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-trusted-ca-bundle\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374431 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374405 5973 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:46:18.374527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374496 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b294cd0-8239-4ec3-ad32-adff86b0f779-tmp\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.374834 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374811 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.375100 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.374819 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-oauth-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.375100 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.375011 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-service-ca\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.375100 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.375094 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b294cd0-8239-4ec3-ad32-adff86b0f779-trusted-ca-bundle\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.376730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.376712 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-oauth-config\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.376833 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.376817 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b294cd0-8239-4ec3-ad32-adff86b0f779-console-serving-cert\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.386608 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.386591 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfds\" (UniqueName: \"kubernetes.io/projected/1b294cd0-8239-4ec3-ad32-adff86b0f779-kube-api-access-lmfds\") pod \"console-699c6fd85b-jjs28\" (UID: \"1b294cd0-8239-4ec3-ad32-adff86b0f779\") " pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.448661 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.448608 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hx8cd" Nov 03 21:46:18.459894 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.459865 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:18.477991 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.477957 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:18.750688 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.750615 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699c6fd85b-jjs28"] Nov 03 21:46:18.753650 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:18.753621 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b294cd0_8239_4ec3_ad32_adff86b0f779.slice/crio-4f27d40c401ab943c8ff53d249960498f79417a4c062d0a02f0eaff436f1bd53 WatchSource:0}: Error finding container 4f27d40c401ab943c8ff53d249960498f79417a4c062d0a02f0eaff436f1bd53: Status 404 returned error can't find the container with id 4f27d40c401ab943c8ff53d249960498f79417a4c062d0a02f0eaff436f1bd53 Nov 03 21:46:18.911515 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:18.911479 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:19.065705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.065624 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:19.453836 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.453802 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c6fd85b-jjs28" event={"ID":"1b294cd0-8239-4ec3-ad32-adff86b0f779","Type":"ContainerStarted","Data":"e39b2f6ec375d95765db57a171fb15a4ac8c7e7599c4c79a407530bbfe07a12e"} Nov 03 21:46:19.454334 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.453852 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699c6fd85b-jjs28" event={"ID":"1b294cd0-8239-4ec3-ad32-adff86b0f779","Type":"ContainerStarted","Data":"4f27d40c401ab943c8ff53d249960498f79417a4c062d0a02f0eaff436f1bd53"} Nov 03 21:46:19.477353 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.477294 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-699c6fd85b-jjs28" podStartSLOduration=1.477281288 podStartE2EDuration="1.477281288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:19.476775609 +0000 UTC m=+19.251625458" watchObservedRunningTime="2026-11-03 21:46:19.477281288 +0000 UTC m=+19.252131135" Nov 03 21:46:19.992012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.991956 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:46:19.994654 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.994636 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:19.998728 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998699 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 21:46:19.998848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998732 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 21:46:19.998848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998739 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 21:46:19.998848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998792 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 21:46:19.998848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998805 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 21:46:19.998848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.998821 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 21:46:19.999950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:19.999931 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 21:46:20.000024 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.000000 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 21:46:20.000149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.000134 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8vqcv\"" Nov 03 21:46:20.004201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.004179 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 21:46:20.014570 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.014551 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:46:20.089626 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089600 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.089626 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089632 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.089854 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089661 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.089854 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089739 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.089961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089875 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.089961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.089921 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090010 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090061 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090047 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090100 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghxd\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-kube-api-access-qghxd\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090169 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090128 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090169 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090157 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-out\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090211 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-web-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.090287 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.090233 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191314 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191289 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191329 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191344 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191367 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191386 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191431 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191490 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191524 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191547 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191577 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghxd\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-kube-api-access-qghxd\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191608 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191640 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-out\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.191683 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.191679 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-web-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.192037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.192007 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.192640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.192569 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.192839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.192740 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.194834 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.194809 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.194935 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.194871 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195225 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-web-config\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195361 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195332 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195519 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195538 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195801 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195777 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.195869 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.195792 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-config-out\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.196882 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.196867 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.207554 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.207537 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghxd\" (UniqueName: \"kubernetes.io/projected/5a3b76f3-a06f-42d5-b2e6-d50d90ef4731-kube-api-access-qghxd\") pod \"alertmanager-main-0\" (UID: \"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.311734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.311672 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 21:46:20.596971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.596891 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 21:46:20.599859 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:46:20.599826 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3b76f3_a06f_42d5_b2e6_d50d90ef4731.slice/crio-b8bdfeb208ce30d85fbfa9902c058db5a3fe41b35c3b86cb79db03b61aca7b1f WatchSource:0}: Error finding container b8bdfeb208ce30d85fbfa9902c058db5a3fe41b35c3b86cb79db03b61aca7b1f: Status 404 returned error can't find the container with id b8bdfeb208ce30d85fbfa9902c058db5a3fe41b35c3b86cb79db03b61aca7b1f Nov 03 21:46:20.845232 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:20.845181 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:21.005657 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:21.005627 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:21.462007 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:21.461981 5973 generic.go:356] "Generic (PLEG): container finished" podID="5a3b76f3-a06f-42d5-b2e6-d50d90ef4731" containerID="d2297e604d67cb2d00ac3ffc57a49b0af9ee57cfae96e6243ae0055b7da7edff" exitCode=0 Nov 03 21:46:21.462176 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:21.462076 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerDied","Data":"d2297e604d67cb2d00ac3ffc57a49b0af9ee57cfae96e6243ae0055b7da7edff"} Nov 03 21:46:21.462248 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:21.462197 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"b8bdfeb208ce30d85fbfa9902c058db5a3fe41b35c3b86cb79db03b61aca7b1f"} Nov 03 21:46:21.463284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:21.463270 5973 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:46:24.476922 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:24.476889 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"7a3964215d1c3215ac89ae1e08493c8dd1ffbc467a71c4257468391d15902202"} Nov 03 21:46:24.477296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:24.476954 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"35168d076a13e96ee7555b93bd55cd19aa6a98620e2c24678ca8c04497f97197"} Nov 03 21:46:24.477296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:24.476980 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"a0c5479f441c6d0679f3a9f2b3c9deafca67ba2f5b6e208baa785e4bd35b8c49"} Nov 03 21:46:24.477296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:24.477004 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"70c6fb9a997166e174279165798f18bf8b2c2068faa275d5d3aabe46effe0d71"} Nov 03 21:46:24.477296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:24.477026 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"9e1982cb749e4d54d4176b9143b753d94498f9dce57124807b1852423be55b51"} Nov 03 21:46:25.483814 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:25.483779 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a3b76f3-a06f-42d5-b2e6-d50d90ef4731","Type":"ContainerStarted","Data":"cc82bf1290f8d5777e968b0141f435334309b9cdce9e1f3e05f1c101946bbef2"} Nov 03 21:46:25.518712 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:25.518665 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.409797078 podStartE2EDuration="6.518647817s" totalImagesPullingTime="3.095619867s" totalInitContainerRuntime="13.230872ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:25.517105777 +0000 UTC m=+25.291955632" watchObservedRunningTime="2026-11-03 21:46:25.518647817 +0000 UTC m=+25.293497665" Nov 03 21:46:26.331945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.331891 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c56dbb448-ftbq5" podUID="a74320bc-ea6d-4469-98e7-7440ba94cef2" containerName="console" containerID="cri-o://1395bb81eaa38bfd170eea5d9aa469a7e0c52c92c8672e1898b3299f74a7c5ce" gracePeriod=15 Nov 03 21:46:26.489695 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.489666 5973 generic.go:356] "Generic (PLEG): container finished" podID="a74320bc-ea6d-4469-98e7-7440ba94cef2" containerID="1395bb81eaa38bfd170eea5d9aa469a7e0c52c92c8672e1898b3299f74a7c5ce" exitCode=0 Nov 03 21:46:26.490064 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.489746 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c56dbb448-ftbq5" event={"ID":"a74320bc-ea6d-4469-98e7-7440ba94cef2","Type":"ContainerDied","Data":"1395bb81eaa38bfd170eea5d9aa469a7e0c52c92c8672e1898b3299f74a7c5ce"} Nov 03 21:46:26.568925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.568903 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:26.656051 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656025 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656079 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656113 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656141 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656163 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656336 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656194 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656336 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656246 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656336 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656303 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") pod \"a74320bc-ea6d-4469-98e7-7440ba94cef2\" (UID: \"a74320bc-ea6d-4469-98e7-7440ba94cef2\") " Nov 03 21:46:26.656594 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656422 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:26.656594 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656444 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:46:26.656768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656599 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:26.656768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656608 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:26.656907 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.656882 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:26.658743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.658710 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:26.658818 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.658742 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "kube-api-access-wq5z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:26.658818 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.658777 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert" pod "a74320bc-ea6d-4469-98e7-7440ba94cef2" (UID: "a74320bc-ea6d-4469-98e7-7440ba94cef2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:26.757743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757716 5973 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-oauth-config\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757739 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5z4\" (UniqueName: \"kubernetes.io/projected/a74320bc-ea6d-4469-98e7-7440ba94cef2-kube-api-access-wq5z4\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757749 5973 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-config\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757758 5973 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74320bc-ea6d-4469-98e7-7440ba94cef2-console-serving-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757769 5973 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-oauth-serving-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757777 5973 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74320bc-ea6d-4469-98e7-7440ba94cef2-tmp\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757785 5973 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-trusted-ca-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:26.757929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:26.757795 5973 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74320bc-ea6d-4469-98e7-7440ba94cef2-service-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:27.494373 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:27.494342 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c56dbb448-ftbq5" event={"ID":"a74320bc-ea6d-4469-98e7-7440ba94cef2","Type":"ContainerDied","Data":"37042f6cb1a3e5a99e0e001d66c908fddf72fa53ffe3d02da525f97a3111f1ea"} Nov 03 21:46:27.494762 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:27.494398 5973 scope.go:122] "RemoveContainer" containerID="1395bb81eaa38bfd170eea5d9aa469a7e0c52c92c8672e1898b3299f74a7c5ce" Nov 03 21:46:27.494762 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:27.494349 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c56dbb448-ftbq5" Nov 03 21:46:27.515681 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:27.515655 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:46:27.524563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:27.524526 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c56dbb448-ftbq5"] Nov 03 21:46:28.346165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.344639 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74320bc-ea6d-4469-98e7-7440ba94cef2" path="/var/lib/kubelet/pods/a74320bc-ea6d-4469-98e7-7440ba94cef2/volumes" Nov 03 21:46:28.478897 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.478865 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:28.479062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.478930 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:28.483610 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.483591 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:28.502034 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.502005 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-699c6fd85b-jjs28" Nov 03 21:46:28.998702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:28.998670 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:29.147385 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:29.147348 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:35.836008 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:35.835971 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:35.982648 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:35.982616 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:36.145110 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:36.145074 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f6h64" Nov 03 21:46:44.398964 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:46:44.398918 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:46:55.666478 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:46:55.666429 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:14.399080 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:47:14.399036 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:47:44.398811 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:47:44.398724 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:47:45.634878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:47:45.634843 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:45.786149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:47:45.786114 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:55.713290 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:47:55.713250 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:57.174215 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:47:57.174176 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:14.397933 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:48:14.397881 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:48:44.398232 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:48:44.398194 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:48:48.398873 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:48.398840 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:50.452315 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:50.452280 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:50.614338 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:50.614302 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:51.199117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.199085 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2"] Nov 03 21:48:51.199420 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.199407 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a74320bc-ea6d-4469-98e7-7440ba94cef2" containerName="console" Nov 03 21:48:51.199488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.199422 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74320bc-ea6d-4469-98e7-7440ba94cef2" containerName="console" Nov 03 21:48:51.199526 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.199509 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74320bc-ea6d-4469-98e7-7440ba94cef2" containerName="console" Nov 03 21:48:51.202554 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.202539 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.207536 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.207516 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:48:51.207729 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.207712 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wgbdf\"" Nov 03 21:48:51.207797 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.207789 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:48:51.223026 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.222997 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2"] Nov 03 21:48:51.311394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.311360 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.311573 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.311405 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.311573 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.311505 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5qk\" (UniqueName: \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.411845 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.411810 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.412003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.411852 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.412003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.411906 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5qk\" (UniqueName: \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.412219 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.412199 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.412275 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.412241 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.424889 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.424863 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5qk\" (UniqueName: \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.517994 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.517912 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:48:51.789584 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.789426 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2"] Nov 03 21:48:51.793182 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:48:51.793156 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0b54f4_b7bb_4057_b257_ef5ebba51256.slice/crio-5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f WatchSource:0}: Error finding container 5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f: Status 404 returned error can't find the container with id 5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f Nov 03 21:48:51.971712 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:51.971680 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" event={"ID":"cc0b54f4-b7bb-4057-b257-ef5ebba51256","Type":"ContainerStarted","Data":"5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f"} Nov 03 21:48:55.769161 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:55.769124 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:57.994684 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:57.994657 5973 generic.go:356] "Generic (PLEG): container finished" podID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerID="f8e7df5c20ef0daa13c02f679673e23870197bb432f2174acc6e597fb393179d" exitCode=0 Nov 03 21:48:57.994985 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:48:57.994716 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" event={"ID":"cc0b54f4-b7bb-4057-b257-ef5ebba51256","Type":"ContainerDied","Data":"f8e7df5c20ef0daa13c02f679673e23870197bb432f2174acc6e597fb393179d"} Nov 03 21:49:01.008056 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:01.008028 5973 generic.go:356] "Generic (PLEG): container finished" podID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerID="39b715ab55ad94ac7a8699975867f7dcc400daaced7583864326728432c674fe" exitCode=0 Nov 03 21:49:01.008385 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:01.008115 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" event={"ID":"cc0b54f4-b7bb-4057-b257-ef5ebba51256","Type":"ContainerDied","Data":"39b715ab55ad94ac7a8699975867f7dcc400daaced7583864326728432c674fe"} Nov 03 21:49:09.039814 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:09.039774 5973 generic.go:356] "Generic (PLEG): container finished" podID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerID="bd67a4aff238caacf05b33d1e5b95c70da7a14cd519ef6655a448f44af748ef2" exitCode=0 Nov 03 21:49:09.040260 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:09.039863 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" event={"ID":"cc0b54f4-b7bb-4057-b257-ef5ebba51256","Type":"ContainerDied","Data":"bd67a4aff238caacf05b33d1e5b95c70da7a14cd519ef6655a448f44af748ef2"} Nov 03 21:49:10.160317 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.160294 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:49:10.274478 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.274404 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\") pod \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " Nov 03 21:49:10.274624 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.274547 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\" (UniqueName: \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\") pod \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " Nov 03 21:49:10.274624 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.274582 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\") pod \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\" (UID: \"cc0b54f4-b7bb-4057-b257-ef5ebba51256\") " Nov 03 21:49:10.275067 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.275044 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle" pod "cc0b54f4-b7bb-4057-b257-ef5ebba51256" (UID: "cc0b54f4-b7bb-4057-b257-ef5ebba51256"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:10.276791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.276765 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk" pod "cc0b54f4-b7bb-4057-b257-ef5ebba51256" (UID: "cc0b54f4-b7bb-4057-b257-ef5ebba51256"). InnerVolumeSpecName "kube-api-access-zt5qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:10.279748 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.279721 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util" pod "cc0b54f4-b7bb-4057-b257-ef5ebba51256" (UID: "cc0b54f4-b7bb-4057-b257-ef5ebba51256"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:10.375354 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.375328 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5qk\" (UniqueName: \"kubernetes.io/projected/cc0b54f4-b7bb-4057-b257-ef5ebba51256-kube-api-access-zt5qk\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:10.375354 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.375351 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:10.375512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:10.375361 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc0b54f4-b7bb-4057-b257-ef5ebba51256-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:11.048131 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:11.048097 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" event={"ID":"cc0b54f4-b7bb-4057-b257-ef5ebba51256","Type":"ContainerDied","Data":"5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f"} Nov 03 21:49:11.048292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:11.048141 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdrcx2" Nov 03 21:49:11.048292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:11.048145 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5845c78bf0ee20eccfdacaf825d0eb5ca5273cb66c5f3a9331d5222b6c9caf9f" Nov 03 21:49:12.532382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:12.532340 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:13.866854 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:13.866813 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:14.343828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.343797 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf"] Nov 03 21:49:14.344080 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344066 5973 cpu_manager.go:414] "containerMap: removing container" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="extract" Nov 03 21:49:14.344126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344081 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="extract" Nov 03 21:49:14.344126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344094 5973 cpu_manager.go:414] "containerMap: removing container" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="util" Nov 03 21:49:14.344126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344098 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="util" Nov 03 21:49:14.344126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344113 5973 cpu_manager.go:414] "containerMap: removing container" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="pull" Nov 03 21:49:14.344126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344118 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="pull" Nov 03 21:49:14.344292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.344174 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0b54f4-b7bb-4057-b257-ef5ebba51256" containerName="extract" Nov 03 21:49:14.347142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.347127 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.351231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.351201 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gbd6z\"" Nov 03 21:49:14.352199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.352182 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 21:49:14.352199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.352186 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:49:14.359759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.359741 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf"] Nov 03 21:49:14.398046 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:49:14.398018 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:49:14.407430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.407410 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rdh\" (UniqueName: \"kubernetes.io/projected/3bdcc28d-db15-469c-9e4f-a351e47f6c33-kube-api-access-z6rdh\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.407510 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.407445 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdcc28d-db15-469c-9e4f-a351e47f6c33-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.508668 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.508635 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rdh\" (UniqueName: \"kubernetes.io/projected/3bdcc28d-db15-469c-9e4f-a351e47f6c33-kube-api-access-z6rdh\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.508668 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.508676 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdcc28d-db15-469c-9e4f-a351e47f6c33-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.509112 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.509095 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdcc28d-db15-469c-9e4f-a351e47f6c33-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.524618 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.524598 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rdh\" (UniqueName: \"kubernetes.io/projected/3bdcc28d-db15-469c-9e4f-a351e47f6c33-kube-api-access-z6rdh\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5dkgf\" (UID: \"3bdcc28d-db15-469c-9e4f-a351e47f6c33\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.658355 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.658333 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" Nov 03 21:49:14.942261 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:14.942103 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf"] Nov 03 21:49:14.945806 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:14.945770 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdcc28d_db15_469c_9e4f_a351e47f6c33.slice/crio-06c7f6445c7542a518446fc7534b9938567791b767c67f90621073616364686a WatchSource:0}: Error finding container 06c7f6445c7542a518446fc7534b9938567791b767c67f90621073616364686a: Status 404 returned error can't find the container with id 06c7f6445c7542a518446fc7534b9938567791b767c67f90621073616364686a Nov 03 21:49:15.063826 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:15.063796 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" event={"ID":"3bdcc28d-db15-469c-9e4f-a351e47f6c33","Type":"ContainerStarted","Data":"06c7f6445c7542a518446fc7534b9938567791b767c67f90621073616364686a"} Nov 03 21:49:15.926746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:15.926713 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:16.081935 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:16.081902 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:17.074577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:17.074529 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" event={"ID":"3bdcc28d-db15-469c-9e4f-a351e47f6c33","Type":"ContainerStarted","Data":"26c62c4bcf1722349aa1e709c9e04aec170ed2afaff5dce73be9ebf3243b9597"} Nov 03 21:49:17.098732 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:17.098686 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5dkgf" podStartSLOduration=1.083830956 podStartE2EDuration="3.098674871s" totalImagesPullingTime="2.014843915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:17.097311356 +0000 UTC m=+196.872161203" watchObservedRunningTime="2026-11-03 21:49:17.098674871 +0000 UTC m=+196.873524708" Nov 03 21:49:17.734116 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:17.734082 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:17.940788 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:17.940753 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:18.481219 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.481187 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz"] Nov 03 21:49:18.484819 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.484795 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.493295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.493277 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:49:18.494257 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.494242 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:49:18.494315 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.494260 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-fxt7s\"" Nov 03 21:49:18.497016 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.496996 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz"] Nov 03 21:49:18.541206 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.541176 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.541338 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.541225 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.541338 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.541322 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4cv\" (UniqueName: \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.591166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.591140 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bp284"] Nov 03 21:49:18.594281 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.594265 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.597963 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.597946 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 21:49:18.598055 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.598014 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 21:49:18.598122 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.598060 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gktzk\"" Nov 03 21:49:18.603871 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.603848 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bp284"] Nov 03 21:49:18.642152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642127 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.642259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642167 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcxf\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-kube-api-access-kgcxf\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.642310 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642284 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.642363 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642315 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.642415 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642373 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4cv\" (UniqueName: \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.642493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642448 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.642687 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.642666 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.655001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.654975 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4cv\" (UniqueName: \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.743386 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.743331 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcxf\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-kube-api-access-kgcxf\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.743386 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.743364 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.755690 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.755670 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.755887 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.755865 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcxf\" (UniqueName: \"kubernetes.io/projected/e10b590f-289d-4866-b886-ac42c4b65f9d-kube-api-access-kgcxf\") pod \"cert-manager-webhook-5dbbbf75cc-bp284\" (UID: \"e10b590f-289d-4866-b886-ac42c4b65f9d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:18.799927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.799906 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:18.914085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:18.914043 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:19.072222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:19.072158 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz"] Nov 03 21:49:19.075253 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:19.075221 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38acba85_79f6_456d_a944_fcc823326c03.slice/crio-0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0 WatchSource:0}: Error finding container 0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0: Status 404 returned error can't find the container with id 0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0 Nov 03 21:49:19.085729 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:19.085705 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" event={"ID":"38acba85-79f6-456d-a944-fcc823326c03","Type":"ContainerStarted","Data":"0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0"} Nov 03 21:49:19.185503 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:19.185477 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bp284"] Nov 03 21:49:19.231247 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:19.231212 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10b590f_289d_4866_b886_ac42c4b65f9d.slice/crio-927d8bed2359f99bb036f82fea8187740c35c46c680f9fab01940047a6e18339 WatchSource:0}: Error finding container 927d8bed2359f99bb036f82fea8187740c35c46c680f9fab01940047a6e18339: Status 404 returned error can't find the container with id 927d8bed2359f99bb036f82fea8187740c35c46c680f9fab01940047a6e18339 Nov 03 21:49:20.090070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:20.090039 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" event={"ID":"e10b590f-289d-4866-b886-ac42c4b65f9d","Type":"ContainerStarted","Data":"927d8bed2359f99bb036f82fea8187740c35c46c680f9fab01940047a6e18339"} Nov 03 21:49:20.091598 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:20.091575 5973 generic.go:356] "Generic (PLEG): container finished" podID="38acba85-79f6-456d-a944-fcc823326c03" containerID="0fb4a7265369190bae4b01f135d261d37066d41ad68065ee05711e83c5356919" exitCode=0 Nov 03 21:49:20.091701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:20.091604 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" event={"ID":"38acba85-79f6-456d-a944-fcc823326c03","Type":"ContainerDied","Data":"0fb4a7265369190bae4b01f135d261d37066d41ad68065ee05711e83c5356919"} Nov 03 21:49:20.113041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:20.113011 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:20.311648 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:20.311619 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:22.722478 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:22.722427 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:22.882332 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:22.882308 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:23.106706 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:23.106678 5973 generic.go:356] "Generic (PLEG): container finished" podID="38acba85-79f6-456d-a944-fcc823326c03" containerID="b8a62caecff153b7f6acc7b604b75229bc94143769ae8dcb6c6092a5c539b608" exitCode=0 Nov 03 21:49:23.106858 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:23.106754 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" event={"ID":"38acba85-79f6-456d-a944-fcc823326c03","Type":"ContainerDied","Data":"b8a62caecff153b7f6acc7b604b75229bc94143769ae8dcb6c6092a5c539b608"} Nov 03 21:49:23.108376 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:23.108349 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" event={"ID":"e10b590f-289d-4866-b886-ac42c4b65f9d","Type":"ContainerStarted","Data":"b433408fc85d572909d1d3011f8e0118ae3b689f689e456151783889a229799e"} Nov 03 21:49:23.108547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:23.108446 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:23.146645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:23.146605 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" podStartSLOduration=1.58929148 podStartE2EDuration="5.146591787s" totalImagesPullingTime="3.557300307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:23.146538728 +0000 UTC m=+202.921388575" watchObservedRunningTime="2026-11-03 21:49:23.146591787 +0000 UTC m=+202.921441634" Nov 03 21:49:24.123247 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:24.123215 5973 generic.go:356] "Generic (PLEG): container finished" podID="38acba85-79f6-456d-a944-fcc823326c03" containerID="a3be3b1a0fe21f71f6c5610cf72036fb4ccc8ebc863b0f451b92c11be7c0e7db" exitCode=0 Nov 03 21:49:24.123738 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:24.123365 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" event={"ID":"38acba85-79f6-456d-a944-fcc823326c03","Type":"ContainerDied","Data":"a3be3b1a0fe21f71f6c5610cf72036fb4ccc8ebc863b0f451b92c11be7c0e7db"} Nov 03 21:49:25.065689 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.065648 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:25.257758 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.257729 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:25.401245 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.401212 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\" (UniqueName: \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\") pod \"38acba85-79f6-456d-a944-fcc823326c03\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " Nov 03 21:49:25.401406 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.401287 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\") pod \"38acba85-79f6-456d-a944-fcc823326c03\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " Nov 03 21:49:25.401406 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.401344 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\") pod \"38acba85-79f6-456d-a944-fcc823326c03\" (UID: \"38acba85-79f6-456d-a944-fcc823326c03\") " Nov 03 21:49:25.401761 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.401735 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle" pod "38acba85-79f6-456d-a944-fcc823326c03" (UID: "38acba85-79f6-456d-a944-fcc823326c03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:25.403397 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.403376 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv" pod "38acba85-79f6-456d-a944-fcc823326c03" (UID: "38acba85-79f6-456d-a944-fcc823326c03"). InnerVolumeSpecName "kube-api-access-6h4cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:25.406149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.406117 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util" pod "38acba85-79f6-456d-a944-fcc823326c03" (UID: "38acba85-79f6-456d-a944-fcc823326c03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:25.502113 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.502087 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:25.502113 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.502110 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4cv\" (UniqueName: \"kubernetes.io/projected/38acba85-79f6-456d-a944-fcc823326c03-kube-api-access-6h4cv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:25.502280 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:25.502122 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38acba85-79f6-456d-a944-fcc823326c03-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:26.138538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:26.138502 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" event={"ID":"38acba85-79f6-456d-a944-fcc823326c03","Type":"ContainerDied","Data":"0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0"} Nov 03 21:49:26.138707 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:26.138555 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fba2ae6cb97a4923657917d81fc04c6bc180be085eb81049586d7961a2325c0" Nov 03 21:49:26.138707 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:26.138518 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kzpzz" Nov 03 21:49:27.750051 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:27.750023 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:28.006643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006576 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-2mwtk"] Nov 03 21:49:28.006878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006867 5973 cpu_manager.go:414] "containerMap: removing container" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="pull" Nov 03 21:49:28.006916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006880 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="pull" Nov 03 21:49:28.006916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006897 5973 cpu_manager.go:414] "containerMap: removing container" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="util" Nov 03 21:49:28.006916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006902 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="util" Nov 03 21:49:28.006916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006912 5973 cpu_manager.go:414] "containerMap: removing container" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="extract" Nov 03 21:49:28.007032 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006918 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="extract" Nov 03 21:49:28.007032 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.006960 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="38acba85-79f6-456d-a944-fcc823326c03" containerName="extract" Nov 03 21:49:28.010776 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.010760 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.019018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.018997 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f9bv4\"" Nov 03 21:49:28.020079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.020058 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2mwtk"] Nov 03 21:49:28.121629 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.121597 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lj9\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-kube-api-access-p2lj9\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.121759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.121651 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-bound-sa-token\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.222940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.222912 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lj9\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-kube-api-access-p2lj9\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.223079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.222954 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-bound-sa-token\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.235919 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.235891 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-bound-sa-token\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.235919 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.235916 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lj9\" (UniqueName: \"kubernetes.io/projected/90da377f-ec22-4ebc-bb67-410d5e16954b-kube-api-access-p2lj9\") pod \"cert-manager-597b95cb84-2mwtk\" (UID: \"90da377f-ec22-4ebc-bb67-410d5e16954b\") " pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.323269 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.323191 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2mwtk" Nov 03 21:49:28.601843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:28.601738 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2mwtk"] Nov 03 21:49:28.605706 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:28.605674 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90da377f_ec22_4ebc_bb67_410d5e16954b.slice/crio-64aa5b94fd306a5e2147697a84117a653129b474705e6a9bff266f12ab9778a1 WatchSource:0}: Error finding container 64aa5b94fd306a5e2147697a84117a653129b474705e6a9bff266f12ab9778a1: Status 404 returned error can't find the container with id 64aa5b94fd306a5e2147697a84117a653129b474705e6a9bff266f12ab9778a1 Nov 03 21:49:29.126044 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:29.126019 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bp284" Nov 03 21:49:29.157050 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:29.157013 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2mwtk" event={"ID":"90da377f-ec22-4ebc-bb67-410d5e16954b","Type":"ContainerStarted","Data":"e17b4113417283a0b5f06f6d073efe0b638eb94b6111abb12844afafd1571bdb"} Nov 03 21:49:29.157219 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:29.157064 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2mwtk" event={"ID":"90da377f-ec22-4ebc-bb67-410d5e16954b","Type":"ContainerStarted","Data":"64aa5b94fd306a5e2147697a84117a653129b474705e6a9bff266f12ab9778a1"} Nov 03 21:49:29.177958 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:29.177905 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-2mwtk" podStartSLOduration=2.177887604 podStartE2EDuration="2.177887604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:29.177440181 +0000 UTC m=+208.952290028" watchObservedRunningTime="2026-11-03 21:49:29.177887604 +0000 UTC m=+208.952737453" Nov 03 21:49:34.472969 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:34.472934 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:34.627930 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:34.627886 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:35.910477 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:35.910425 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:36.061756 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:36.061722 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:41.640314 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:41.640276 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:41.793385 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:41.793345 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:43.158125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.158084 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:43.884037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.884007 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq"] Nov 03 21:49:43.886377 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.886362 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:43.890263 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.890240 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:49:43.891524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.891498 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2tmwh\"" Nov 03 21:49:43.891623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.891543 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:49:43.897255 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:43.897235 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq"] Nov 03 21:49:44.048433 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.048399 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.048433 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.048435 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v5x\" (UniqueName: \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.048667 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.048476 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.148773 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.148748 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.148893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.148780 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v5x\" (UniqueName: \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.148893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.148804 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.149208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.149187 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.149208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.149200 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.161934 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.161903 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v5x\" (UniqueName: \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.199695 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.199673 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:44.398517 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:49:44.398484 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:49:44.471382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:44.471207 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq"] Nov 03 21:49:44.473919 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:44.473887 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2905d0ad_53e0_450a_92b2_4ccc0c819614.slice/crio-da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214 WatchSource:0}: Error finding container da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214: Status 404 returned error can't find the container with id da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214 Nov 03 21:49:45.224547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:45.224516 5973 generic.go:356] "Generic (PLEG): container finished" podID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerID="c2efb89fd987cbe7064c1c7b89835cf6df50affea5ea64489927309522f533df" exitCode=0 Nov 03 21:49:45.224942 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:45.224611 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" event={"ID":"2905d0ad-53e0-450a-92b2-4ccc0c819614","Type":"ContainerDied","Data":"c2efb89fd987cbe7064c1c7b89835cf6df50affea5ea64489927309522f533df"} Nov 03 21:49:45.224942 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:45.224669 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" event={"ID":"2905d0ad-53e0-450a-92b2-4ccc0c819614","Type":"ContainerStarted","Data":"da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214"} Nov 03 21:49:46.230498 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:46.230445 5973 generic.go:356] "Generic (PLEG): container finished" podID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerID="4dbb8e3be3e9ad5d79572c4c42cd63b8e8f3f56b7f2fc49702f79a2587191b4f" exitCode=0 Nov 03 21:49:46.230900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:46.230513 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" event={"ID":"2905d0ad-53e0-450a-92b2-4ccc0c819614","Type":"ContainerDied","Data":"4dbb8e3be3e9ad5d79572c4c42cd63b8e8f3f56b7f2fc49702f79a2587191b4f"} Nov 03 21:49:47.236184 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:47.236150 5973 generic.go:356] "Generic (PLEG): container finished" podID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerID="53573ede0eedfd18ab7b6efc747c7ce10628680cf55e5a73dec5aa34a6f5268c" exitCode=0 Nov 03 21:49:47.236559 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:47.236227 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" event={"ID":"2905d0ad-53e0-450a-92b2-4ccc0c819614","Type":"ContainerDied","Data":"53573ede0eedfd18ab7b6efc747c7ce10628680cf55e5a73dec5aa34a6f5268c"} Nov 03 21:49:48.365882 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.365855 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:48.481792 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.481767 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\" (UniqueName: \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\") pod \"2905d0ad-53e0-450a-92b2-4ccc0c819614\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " Nov 03 21:49:48.481954 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.481847 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\") pod \"2905d0ad-53e0-450a-92b2-4ccc0c819614\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " Nov 03 21:49:48.481954 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.481876 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\") pod \"2905d0ad-53e0-450a-92b2-4ccc0c819614\" (UID: \"2905d0ad-53e0-450a-92b2-4ccc0c819614\") " Nov 03 21:49:48.482364 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.482341 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle" pod "2905d0ad-53e0-450a-92b2-4ccc0c819614" (UID: "2905d0ad-53e0-450a-92b2-4ccc0c819614"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:48.483918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.483887 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x" pod "2905d0ad-53e0-450a-92b2-4ccc0c819614" (UID: "2905d0ad-53e0-450a-92b2-4ccc0c819614"). InnerVolumeSpecName "kube-api-access-m7v5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:48.487418 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.487385 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util" pod "2905d0ad-53e0-450a-92b2-4ccc0c819614" (UID: "2905d0ad-53e0-450a-92b2-4ccc0c819614"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:48.583165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.583107 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:48.583165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.583129 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2905d0ad-53e0-450a-92b2-4ccc0c819614-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:48.583165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:48.583138 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7v5x\" (UniqueName: \"kubernetes.io/projected/2905d0ad-53e0-450a-92b2-4ccc0c819614-kube-api-access-m7v5x\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:49.246516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:49.246487 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" Nov 03 21:49:49.246516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:49.246491 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715twvq" event={"ID":"2905d0ad-53e0-450a-92b2-4ccc0c819614","Type":"ContainerDied","Data":"da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214"} Nov 03 21:49:49.247540 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:49.246538 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da3f53cff079161570d9c291e48defb2ba87cc8ccf21a67310a58b4cbb505214" Nov 03 21:49:54.121676 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.121641 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:54.280101 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.280064 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:54.766298 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.766257 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:54.814670 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.814642 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm"] Nov 03 21:49:54.814971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.814958 5973 cpu_manager.go:414] "containerMap: removing container" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="util" Nov 03 21:49:54.815023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.814972 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="util" Nov 03 21:49:54.815023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.814983 5973 cpu_manager.go:414] "containerMap: removing container" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="pull" Nov 03 21:49:54.815023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.814987 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="pull" Nov 03 21:49:54.815023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.815003 5973 cpu_manager.go:414] "containerMap: removing container" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="extract" Nov 03 21:49:54.815023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.815007 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="extract" Nov 03 21:49:54.815188 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.815047 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="2905d0ad-53e0-450a-92b2-4ccc0c819614" containerName="extract" Nov 03 21:49:54.817014 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.816996 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.820772 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.820753 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 21:49:54.828705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.828672 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 21:49:54.828705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.828679 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 21:49:54.829648 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.829613 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c5jhj\"" Nov 03 21:49:54.829750 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.829681 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Nov 03 21:49:54.830117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830087 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-tmp\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.830405 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830334 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-manifests\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.830505 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830488 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7bv\" (UniqueName: \"kubernetes.io/projected/799bbffe-80a5-41be-a49e-8dd63c8bfb18-kube-api-access-db7bv\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.830583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830552 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-metrics-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.830643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830598 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-webhook-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.830643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.830619 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.853243 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.853221 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm"] Nov 03 21:49:54.921200 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.921179 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 21:49:54.931616 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931590 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-tmp\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.931734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931622 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-manifests\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.931734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931639 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7bv\" (UniqueName: \"kubernetes.io/projected/799bbffe-80a5-41be-a49e-8dd63c8bfb18-kube-api-access-db7bv\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.931734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931668 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-metrics-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.931892 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931793 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-webhook-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.931892 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.931836 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.932100 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.932077 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-manifests\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.934017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.933997 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799bbffe-80a5-41be-a49e-8dd63c8bfb18-tmp\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.934171 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.934154 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-metrics-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.934334 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.934314 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-webhook-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.934369 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.934314 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799bbffe-80a5-41be-a49e-8dd63c8bfb18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:54.943904 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:54.943885 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7bv\" (UniqueName: \"kubernetes.io/projected/799bbffe-80a5-41be-a49e-8dd63c8bfb18-kube-api-access-db7bv\") pod \"opendatahub-operator-controller-manager-77f6458f94-p87pm\" (UID: \"799bbffe-80a5-41be-a49e-8dd63c8bfb18\") " pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:55.028046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.027968 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.129424 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.129393 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:49:55.183094 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.183067 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.341282 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.341179 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.427020 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.426993 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm"] Nov 03 21:49:55.429545 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:49:55.429515 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799bbffe_80a5_41be_a49e_8dd63c8bfb18.slice/crio-23f1e0095789b3be1300e29e7cfa1fd7da2427a16ccd2dd3e6e78b3d51d554b7 WatchSource:0}: Error finding container 23f1e0095789b3be1300e29e7cfa1fd7da2427a16ccd2dd3e6e78b3d51d554b7: Status 404 returned error can't find the container with id 23f1e0095789b3be1300e29e7cfa1fd7da2427a16ccd2dd3e6e78b3d51d554b7 Nov 03 21:49:55.571399 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.571368 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.739103 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.739066 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:55.899852 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:55.899821 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:56.275152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:49:56.275115 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" event={"ID":"799bbffe-80a5-41be-a49e-8dd63c8bfb18","Type":"ContainerStarted","Data":"23f1e0095789b3be1300e29e7cfa1fd7da2427a16ccd2dd3e6e78b3d51d554b7"} Nov 03 21:50:01.304660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:01.304627 5973 generic.go:356] "Generic (PLEG): container finished" podID="799bbffe-80a5-41be-a49e-8dd63c8bfb18" containerID="6d114b23e4898da93394521e288a13b9dafda059ab2414a4ac79a122932d2368" exitCode=0 Nov 03 21:50:01.304995 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:01.304717 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" event={"ID":"799bbffe-80a5-41be-a49e-8dd63c8bfb18","Type":"ContainerDied","Data":"6d114b23e4898da93394521e288a13b9dafda059ab2414a4ac79a122932d2368"} Nov 03 21:50:01.503195 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:01.503161 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:01.713690 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:01.713657 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:02.311382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:02.311339 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" event={"ID":"799bbffe-80a5-41be-a49e-8dd63c8bfb18","Type":"ContainerStarted","Data":"7217574d4745f41d36ce192d62c66cf869aa0cdcbf5771ca77f56be89573dfdd"} Nov 03 21:50:02.311847 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:02.311505 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:50:02.355430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:02.355383 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" podStartSLOduration=2.173067052 podStartE2EDuration="8.355369622s" totalImagesPullingTime="5.701277732s" totalInitContainerRuntime="481.024838ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:02.354112694 +0000 UTC m=+242.128962541" watchObservedRunningTime="2026-11-03 21:50:02.355369622 +0000 UTC m=+242.130219468" Nov 03 21:50:13.317721 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:13.317688 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77f6458f94-p87pm" Nov 03 21:50:14.398636 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:50:14.398597 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:50:15.511399 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.511368 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:15.652404 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.652373 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65998dd86f-c9x59"] Nov 03 21:50:15.655564 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.655549 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.663182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.663162 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 21:50:15.663720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.663692 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8z4qx\"" Nov 03 21:50:15.663720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.663691 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 21:50:15.663720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.663716 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 21:50:15.663860 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.663716 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 21:50:15.679279 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.679249 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65998dd86f-c9x59"] Nov 03 21:50:15.683968 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.683937 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bda906-6e41-4c07-bf43-ea992687d417-tls-certs\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.684077 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.684015 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bda906-6e41-4c07-bf43-ea992687d417-tmp\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.684077 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.684041 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5cc\" (UniqueName: \"kubernetes.io/projected/95bda906-6e41-4c07-bf43-ea992687d417-kube-api-access-5d5cc\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.697228 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.697205 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:15.785261 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.785173 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bda906-6e41-4c07-bf43-ea992687d417-tls-certs\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.785261 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.785205 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bda906-6e41-4c07-bf43-ea992687d417-tmp\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.785261 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.785225 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5cc\" (UniqueName: \"kubernetes.io/projected/95bda906-6e41-4c07-bf43-ea992687d417-kube-api-access-5d5cc\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.787744 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.787722 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bda906-6e41-4c07-bf43-ea992687d417-tmp\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.787958 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.787939 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bda906-6e41-4c07-bf43-ea992687d417-tls-certs\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.798796 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.798771 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5cc\" (UniqueName: \"kubernetes.io/projected/95bda906-6e41-4c07-bf43-ea992687d417-kube-api-access-5d5cc\") pod \"kube-auth-proxy-65998dd86f-c9x59\" (UID: \"95bda906-6e41-4c07-bf43-ea992687d417\") " pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:15.868691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.868649 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:15.967328 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:15.967306 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" Nov 03 21:50:16.239739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:16.239712 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65998dd86f-c9x59"] Nov 03 21:50:16.242093 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:50:16.242062 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bda906_6e41_4c07_bf43_ea992687d417.slice/crio-d4b51882ff413eca34d667e31fcc3fcf3d19dbc4ea85f185230ac87b447db7f1 WatchSource:0}: Error finding container d4b51882ff413eca34d667e31fcc3fcf3d19dbc4ea85f185230ac87b447db7f1: Status 404 returned error can't find the container with id d4b51882ff413eca34d667e31fcc3fcf3d19dbc4ea85f185230ac87b447db7f1 Nov 03 21:50:16.367551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:16.367521 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" event={"ID":"95bda906-6e41-4c07-bf43-ea992687d417","Type":"ContainerStarted","Data":"d4b51882ff413eca34d667e31fcc3fcf3d19dbc4ea85f185230ac87b447db7f1"} Nov 03 21:50:17.051031 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.050999 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:17.192594 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.192524 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-j5t6s"] Nov 03 21:50:17.197393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.196747 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.208238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.208205 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Nov 03 21:50:17.208436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.208416 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-7vxgb\"" Nov 03 21:50:17.210189 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.210097 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-j5t6s"] Nov 03 21:50:17.296826 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.296795 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/53306a2c-5996-4ad9-bd5a-631945b18bec-webhook-certs\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.296981 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.296862 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5rv\" (UniqueName: \"kubernetes.io/projected/53306a2c-5996-4ad9-bd5a-631945b18bec-kube-api-access-xr5rv\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.304119 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.304051 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:17.397368 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.397335 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/53306a2c-5996-4ad9-bd5a-631945b18bec-webhook-certs\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.397662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.397418 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5rv\" (UniqueName: \"kubernetes.io/projected/53306a2c-5996-4ad9-bd5a-631945b18bec-kube-api-access-xr5rv\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.400494 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.400445 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/53306a2c-5996-4ad9-bd5a-631945b18bec-webhook-certs\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.410019 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.409991 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:50:17.414228 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.413870 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.421237 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.421218 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:50:17.421760 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.421735 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5rv\" (UniqueName: \"kubernetes.io/projected/53306a2c-5996-4ad9-bd5a-631945b18bec-kube-api-access-xr5rv\") pod \"odh-observability-584f99c788-j5t6s\" (UID: \"53306a2c-5996-4ad9-bd5a-631945b18bec\") " pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.422990 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.422971 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 21:50:17.423607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.423582 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 21:50:17.423714 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.423695 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7h8b9\"" Nov 03 21:50:17.423775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.423742 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 21:50:17.423826 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.423782 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 21:50:17.472194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.472162 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:17.498200 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498169 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5bj\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498326 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498234 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498326 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498287 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498326 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498317 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498472 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498354 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498472 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498378 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.498472 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.498420 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.516834 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.516804 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:17.599493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599345 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599393 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599434 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599482 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599823 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599514 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599823 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599557 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5bj\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.599823 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.599607 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.600066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.600047 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.605838 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.605815 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.605954 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.605867 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.606076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.606052 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.606146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.606123 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.612028 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.612008 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.612277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.612257 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5bj\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\") pod \"istiod-openshift-gateway-7876bfcb74-cfv4w\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.624934 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.624906 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:17.736846 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.736814 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:17.799075 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:17.799044 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-j5t6s"] Nov 03 21:50:17.809285 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:50:17.809001 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53306a2c_5996_4ad9_bd5a_631945b18bec.slice/crio-b2b2c9230e5a782fbb107faed83f83f5add124d4402afc2aaac417a4900bfd80 WatchSource:0}: Error finding container b2b2c9230e5a782fbb107faed83f83f5add124d4402afc2aaac417a4900bfd80: Status 404 returned error can't find the container with id b2b2c9230e5a782fbb107faed83f83f5add124d4402afc2aaac417a4900bfd80 Nov 03 21:50:18.030931 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:18.030901 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:50:18.032581 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:50:18.032554 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9b5596_e981_49f7_bf4b_680ab246f025.slice/crio-0b7730298adf4be312ecbcaef04432e19d21ce41388f304fed30451de35a105e WatchSource:0}: Error finding container 0b7730298adf4be312ecbcaef04432e19d21ce41388f304fed30451de35a105e: Status 404 returned error can't find the container with id 0b7730298adf4be312ecbcaef04432e19d21ce41388f304fed30451de35a105e Nov 03 21:50:18.379194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:18.379109 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" event={"ID":"0e9b5596-e981-49f7-bf4b-680ab246f025","Type":"ContainerStarted","Data":"0b7730298adf4be312ecbcaef04432e19d21ce41388f304fed30451de35a105e"} Nov 03 21:50:18.380296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:18.380273 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-j5t6s" event={"ID":"53306a2c-5996-4ad9-bd5a-631945b18bec","Type":"ContainerStarted","Data":"b2b2c9230e5a782fbb107faed83f83f5add124d4402afc2aaac417a4900bfd80"} Nov 03 21:50:18.757278 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:18.757237 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:19.388628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:19.388591 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" event={"ID":"95bda906-6e41-4c07-bf43-ea992687d417","Type":"ContainerStarted","Data":"ce548a5ddf781197054d7e5fc15ced8c72444f107f8340650eb8ab7b06f91a35"} Nov 03 21:50:19.409159 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:19.409098 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65998dd86f-c9x59" podStartSLOduration=1.88113328 podStartE2EDuration="4.409084259s" totalImagesPullingTime="2.527950979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:19.408290019 +0000 UTC m=+259.183139866" watchObservedRunningTime="2026-11-03 21:50:19.409084259 +0000 UTC m=+259.183934107" Nov 03 21:50:21.066451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.066420 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:21.216655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.216620 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:21.398452 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.398416 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-j5t6s" event={"ID":"53306a2c-5996-4ad9-bd5a-631945b18bec","Type":"ContainerStarted","Data":"515bfefd49a9010aa329b4ba1502a4abfe12b72a3d65d16a08d6e8e3743088d6"} Nov 03 21:50:21.398618 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.398603 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:21.418259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.418206 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-j5t6s" podStartSLOduration=1.194471126 podStartE2EDuration="4.418194137s" totalImagesPullingTime="3.223723011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:21.417398222 +0000 UTC m=+261.192248062" watchObservedRunningTime="2026-11-03 21:50:21.418194137 +0000 UTC m=+261.193043975" Nov 03 21:50:21.420669 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.420635 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:50:21.420766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.420697 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:50:21.914301 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:21.914270 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:22.074869 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.074827 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:22.403720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.403685 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" event={"ID":"0e9b5596-e981-49f7-bf4b-680ab246f025","Type":"ContainerStarted","Data":"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c"} Nov 03 21:50:22.404160 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.404143 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:22.405531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.405507 5973 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cfv4w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:50:22.405636 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.405565 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:50:22.439908 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.439865 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" podStartSLOduration=2.054859208 podStartE2EDuration="5.439849198s" totalImagesPullingTime="3.38498999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:22.438212774 +0000 UTC m=+262.213062622" watchObservedRunningTime="2026-11-03 21:50:22.439849198 +0000 UTC m=+262.214699046" Nov 03 21:50:22.799235 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.799139 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:22.982820 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:22.982788 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:23.409018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:23.408992 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:50:24.746570 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:24.746524 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:27.900240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:27.900203 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:28.052690 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:28.052655 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:32.405593 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:32.405566 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-j5t6s" Nov 03 21:50:44.398903 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:50:44.398824 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:50:55.888247 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:50:55.888206 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:13.467544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:13.467507 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:13.629205 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:13.629168 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:13.797852 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:13.797767 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:13.954056 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:13.954009 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:14.399248 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:51:14.399211 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:51:14.706363 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:14.706269 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:14.895488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:14.895441 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:15.057005 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:15.056892 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:19.024601 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.024563 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:51:19.025088 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.024777 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerName="discovery" containerID="cri-o://5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c" gracePeriod=30 Nov 03 21:51:19.267730 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.267705 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:51:19.292260 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292170 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292260 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292213 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292261 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292308 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292494 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292661 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292575 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.292661 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.292614 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\") pod \"0e9b5596-e981-49f7-bf4b-680ab246f025\" (UID: \"0e9b5596-e981-49f7-bf4b-680ab246f025\") " Nov 03 21:51:19.293487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.293390 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:19.296020 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.295995 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:19.296150 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.296098 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "kube-api-access-vz5bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:19.296573 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.296538 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:19.296691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.296590 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:19.296691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.296637 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:19.296808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.296739 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig" pod "0e9b5596-e981-49f7-bf4b-680ab246f025" (UID: "0e9b5596-e981-49f7-bf4b-680ab246f025"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:19.393705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393668 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393697 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5bj\" (UniqueName: \"kubernetes.io/projected/0e9b5596-e981-49f7-bf4b-680ab246f025-kube-api-access-vz5bj\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393708 5973 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-dns-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393718 5973 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-csr-ca-configmap\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393727 5973 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-istio-kubeconfig\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393736 5973 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9b5596-e981-49f7-bf4b-680ab246f025-local-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.393971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.393744 5973 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e9b5596-e981-49f7-bf4b-680ab246f025-cacerts\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:51:19.651575 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.651536 5973 generic.go:356] "Generic (PLEG): container finished" podID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerID="5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c" exitCode=0 Nov 03 21:51:19.651770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.651583 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" Nov 03 21:51:19.651770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.651630 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" event={"ID":"0e9b5596-e981-49f7-bf4b-680ab246f025","Type":"ContainerDied","Data":"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c"} Nov 03 21:51:19.651770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.651690 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w" event={"ID":"0e9b5596-e981-49f7-bf4b-680ab246f025","Type":"ContainerDied","Data":"0b7730298adf4be312ecbcaef04432e19d21ce41388f304fed30451de35a105e"} Nov 03 21:51:19.651770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.651716 5973 scope.go:122] "RemoveContainer" containerID="5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c" Nov 03 21:51:19.660918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.660900 5973 scope.go:122] "RemoveContainer" containerID="5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c" Nov 03 21:51:19.661212 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:51:19.661193 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c\": container with ID starting with 5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c not found: ID does not exist" containerID="5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c" Nov 03 21:51:19.661274 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.661222 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c"} err="failed to get container status \"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c\": rpc error: code = NotFound desc = could not find container \"5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c\": container with ID starting with 5ce050c7c2c3c46c65d7406c1cbe4692d2d5b6d8c3651babb14519357a5f303c not found: ID does not exist" Nov 03 21:51:19.674399 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.674375 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:51:19.678655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:19.678632 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cfv4w"] Nov 03 21:51:20.343966 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:20.343936 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" path="/var/lib/kubelet/pods/0e9b5596-e981-49f7-bf4b-680ab246f025/volumes" Nov 03 21:51:22.739504 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:22.739443 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:23.539975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.539940 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn"] Nov 03 21:51:23.540270 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.540258 5973 cpu_manager.go:414] "containerMap: removing container" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerName="discovery" Nov 03 21:51:23.540312 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.540271 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerName="discovery" Nov 03 21:51:23.540345 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.540329 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9b5596-e981-49f7-bf4b-680ab246f025" containerName="discovery" Nov 03 21:51:23.544765 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.544743 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.553576 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.553550 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Nov 03 21:51:23.554050 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554023 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Nov 03 21:51:23.554145 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554038 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Nov 03 21:51:23.554145 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554090 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Nov 03 21:51:23.554348 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554326 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:51:23.554444 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554415 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-srjhq\"" Nov 03 21:51:23.554963 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.554938 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn"] Nov 03 21:51:23.628804 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.628763 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-manager-config\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.629004 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.628816 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txk5v\" (UniqueName: \"kubernetes.io/projected/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-kube-api-access-txk5v\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.629004 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.628907 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.629004 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.628965 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-metrics-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.729831 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.729797 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk5v\" (UniqueName: \"kubernetes.io/projected/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-kube-api-access-txk5v\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.730027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.729842 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.730027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.729869 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-metrics-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.730027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.729902 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-manager-config\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.730574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.730524 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-manager-config\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.732487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.732448 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.732588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.732499 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-metrics-cert\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.742949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.742927 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk5v\" (UniqueName: \"kubernetes.io/projected/22b5f707-a41f-43b1-8ea2-1ad22e4f1934-kube-api-access-txk5v\") pod \"lws-controller-manager-6c6f488f88-q4rdn\" (UID: \"22b5f707-a41f-43b1-8ea2-1ad22e4f1934\") " pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:23.856439 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:23.856354 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:24.130644 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:24.130620 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn"] Nov 03 21:51:24.132308 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:51:24.132284 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b5f707_a41f_43b1_8ea2_1ad22e4f1934.slice/crio-6aab391d89c55b9dceadf0df99f92be4260d92af11d2e90cb0bbea63889017e7 WatchSource:0}: Error finding container 6aab391d89c55b9dceadf0df99f92be4260d92af11d2e90cb0bbea63889017e7: Status 404 returned error can't find the container with id 6aab391d89c55b9dceadf0df99f92be4260d92af11d2e90cb0bbea63889017e7 Nov 03 21:51:24.134371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:24.134354 5973 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:51:24.673710 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:24.673679 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" event={"ID":"22b5f707-a41f-43b1-8ea2-1ad22e4f1934","Type":"ContainerStarted","Data":"6aab391d89c55b9dceadf0df99f92be4260d92af11d2e90cb0bbea63889017e7"} Nov 03 21:51:27.687675 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:27.687635 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" event={"ID":"22b5f707-a41f-43b1-8ea2-1ad22e4f1934","Type":"ContainerStarted","Data":"3594577a2a749818a195c1ed0b5ccaa821d7a428a024aa79cc59ae66d44e37f8"} Nov 03 21:51:27.688075 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:27.687765 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:27.706718 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:27.706630 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" podStartSLOduration=1.401302737 podStartE2EDuration="4.706615275s" totalImagesPullingTime="3.305312538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:27.705500411 +0000 UTC m=+327.480350258" watchObservedRunningTime="2026-11-03 21:51:27.706615275 +0000 UTC m=+327.481465122" Nov 03 21:51:38.693856 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:38.693827 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c6f488f88-q4rdn" Nov 03 21:51:44.399730 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:51:44.399696 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:51:55.941323 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:51:55.941285 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:09.097884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:09.097851 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:09.251685 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:09.251642 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:11.769977 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.769905 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx"] Nov 03 21:52:11.773368 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.773347 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.777130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.777108 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:52:11.778316 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.778297 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:52:11.778412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.778325 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h9qwv\"" Nov 03 21:52:11.782981 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.782956 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx"] Nov 03 21:52:11.830058 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.830036 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.830155 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.830081 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.830155 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.830105 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zsd9\" (UniqueName: \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.860527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.860507 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4"] Nov 03 21:52:11.863745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.863730 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:11.868447 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.868425 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mcbxd\"" Nov 03 21:52:11.873518 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.873499 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4"] Nov 03 21:52:11.930763 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.930740 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2w8p\" (UniqueName: \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:11.930909 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.930778 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.930909 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.930810 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zsd9\" (UniqueName: \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.931018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.930926 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:11.931018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.930970 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:11.931089 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.931044 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.931289 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.931272 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.931326 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.931310 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.942989 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.942966 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zsd9\" (UniqueName: \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:11.960565 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.960548 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz"] Nov 03 21:52:11.964116 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.964096 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:11.967417 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.967396 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r2cb6\"" Nov 03 21:52:11.974121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:11.974104 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz"] Nov 03 21:52:12.031449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031388 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.031449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031417 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.031449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031445 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.031687 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031510 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2w8p\" (UniqueName: \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.031687 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031613 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.031687 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031655 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvsls\" (UniqueName: \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.031851 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031815 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.031911 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.031890 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.046165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.046145 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2w8p\" (UniqueName: \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.059790 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.059767 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g"] Nov 03 21:52:12.063188 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.063174 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.067862 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.067845 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6ptlr\"" Nov 03 21:52:12.076949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.076930 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g"] Nov 03 21:52:12.089583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.089564 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:12.132635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132610 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.132859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132654 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl68\" (UniqueName: \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.132859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132687 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.132859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132720 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.133012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132894 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvsls\" (UniqueName: \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.133012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.132974 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.133093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.133030 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.133130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.133097 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.145383 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.145353 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvsls\" (UniqueName: \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.180230 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.180206 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:12.234027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.233993 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.234244 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.234087 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl68\" (UniqueName: \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.234244 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.234121 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.234751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.234594 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.234751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.234635 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.247232 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.247210 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl68\" (UniqueName: \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.282245 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.282184 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:12.366058 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.366034 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx"] Nov 03 21:52:12.368476 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:52:12.368436 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ffa7f1_5a95_4240_b6c3_09d434bde701.slice/crio-927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398 WatchSource:0}: Error finding container 927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398: Status 404 returned error can't find the container with id 927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398 Nov 03 21:52:12.378975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.378958 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:12.460364 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.460335 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4"] Nov 03 21:52:12.461468 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:52:12.461428 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154b97d3_8aa4_4c4e_ab1f_ff1fa2309ebe.slice/crio-357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487 WatchSource:0}: Error finding container 357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487: Status 404 returned error can't find the container with id 357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487 Nov 03 21:52:12.554555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.554517 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz"] Nov 03 21:52:12.555886 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:52:12.555860 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4248bdd8_6e9f_4595_bf1a_6bf82945c3eb.slice/crio-bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b WatchSource:0}: Error finding container bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b: Status 404 returned error can't find the container with id bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b Nov 03 21:52:12.654654 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.654626 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g"] Nov 03 21:52:12.657528 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:52:12.657500 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193d100c_0a0f_4568_9705_905ea314479e.slice/crio-4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1 WatchSource:0}: Error finding container 4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1: Status 404 returned error can't find the container with id 4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1 Nov 03 21:52:12.877710 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.877613 5973 generic.go:356] "Generic (PLEG): container finished" podID="193d100c-0a0f-4568-9705-905ea314479e" containerID="4105bf48a3557643a7716852d1ee691fd0dad4ac5fcff81a2a4908f8658eec3c" exitCode=0 Nov 03 21:52:12.878130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.877709 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" event={"ID":"193d100c-0a0f-4568-9705-905ea314479e","Type":"ContainerDied","Data":"4105bf48a3557643a7716852d1ee691fd0dad4ac5fcff81a2a4908f8658eec3c"} Nov 03 21:52:12.878130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.877761 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" event={"ID":"193d100c-0a0f-4568-9705-905ea314479e","Type":"ContainerStarted","Data":"4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1"} Nov 03 21:52:12.879290 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.879269 5973 generic.go:356] "Generic (PLEG): container finished" podID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerID="2d96887b070739d96912c38c92a7c32af3097a58ffca8aa4a8d9b37315ad9f00" exitCode=0 Nov 03 21:52:12.879437 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.879349 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" event={"ID":"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe","Type":"ContainerDied","Data":"2d96887b070739d96912c38c92a7c32af3097a58ffca8aa4a8d9b37315ad9f00"} Nov 03 21:52:12.879437 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.879393 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" event={"ID":"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe","Type":"ContainerStarted","Data":"357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487"} Nov 03 21:52:12.881173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.881112 5973 generic.go:356] "Generic (PLEG): container finished" podID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerID="b08162ee2e6fb93ea8e2ae13d00de025bf3388a87923c061989358193b1e71dd" exitCode=0 Nov 03 21:52:12.881240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.881194 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" event={"ID":"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb","Type":"ContainerDied","Data":"b08162ee2e6fb93ea8e2ae13d00de025bf3388a87923c061989358193b1e71dd"} Nov 03 21:52:12.881294 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.881247 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" event={"ID":"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb","Type":"ContainerStarted","Data":"bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b"} Nov 03 21:52:12.883136 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.883119 5973 generic.go:356] "Generic (PLEG): container finished" podID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerID="4d789752e0d8c8fb6d2e1e2078f4809f6c4076344f26aa23f7ae547bd000dbaa" exitCode=0 Nov 03 21:52:12.883216 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.883146 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerDied","Data":"4d789752e0d8c8fb6d2e1e2078f4809f6c4076344f26aa23f7ae547bd000dbaa"} Nov 03 21:52:12.883216 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:12.883181 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerStarted","Data":"927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398"} Nov 03 21:52:13.891013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:13.890977 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerStarted","Data":"33e4d490f4e6c0d9a208acd50bbfa563304f1feb5bd656f2ee9551275481c23d"} Nov 03 21:52:14.397853 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:52:14.397820 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:52:14.898813 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.898776 5973 generic.go:356] "Generic (PLEG): container finished" podID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerID="604d03586edf2b9b0c300105f673a958626f7bf1b00181116df2e83587df58cc" exitCode=0 Nov 03 21:52:14.899250 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.898857 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" event={"ID":"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb","Type":"ContainerDied","Data":"604d03586edf2b9b0c300105f673a958626f7bf1b00181116df2e83587df58cc"} Nov 03 21:52:14.900728 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.900706 5973 generic.go:356] "Generic (PLEG): container finished" podID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerID="33e4d490f4e6c0d9a208acd50bbfa563304f1feb5bd656f2ee9551275481c23d" exitCode=0 Nov 03 21:52:14.900830 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.900802 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerDied","Data":"33e4d490f4e6c0d9a208acd50bbfa563304f1feb5bd656f2ee9551275481c23d"} Nov 03 21:52:14.903142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.903113 5973 generic.go:356] "Generic (PLEG): container finished" podID="193d100c-0a0f-4568-9705-905ea314479e" containerID="577d0076d9b7a24b25613191f9d7dd707263e12b6b3c7b13fa57282ae515cc81" exitCode=0 Nov 03 21:52:14.903244 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.903197 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" event={"ID":"193d100c-0a0f-4568-9705-905ea314479e","Type":"ContainerDied","Data":"577d0076d9b7a24b25613191f9d7dd707263e12b6b3c7b13fa57282ae515cc81"} Nov 03 21:52:14.905107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.905085 5973 generic.go:356] "Generic (PLEG): container finished" podID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerID="41e40fa5031b690c976d65ae95eef60a745d9af9241121f85d98db666f4d9ef2" exitCode=0 Nov 03 21:52:14.905199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:14.905169 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" event={"ID":"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe","Type":"ContainerDied","Data":"41e40fa5031b690c976d65ae95eef60a745d9af9241121f85d98db666f4d9ef2"} Nov 03 21:52:15.912133 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.912098 5973 generic.go:356] "Generic (PLEG): container finished" podID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerID="5f36ad9842af9bc92b52f9e785e0c3e77bbaab6b27ad384b6ff19c5175201ac6" exitCode=0 Nov 03 21:52:15.912528 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.912170 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" event={"ID":"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb","Type":"ContainerDied","Data":"5f36ad9842af9bc92b52f9e785e0c3e77bbaab6b27ad384b6ff19c5175201ac6"} Nov 03 21:52:15.914278 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.914250 5973 generic.go:356] "Generic (PLEG): container finished" podID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerID="a3640a5f0be80cbbde8d98ab9f3147675a43ca887b2d0f2b786049141f561630" exitCode=0 Nov 03 21:52:15.914393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.914283 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerDied","Data":"a3640a5f0be80cbbde8d98ab9f3147675a43ca887b2d0f2b786049141f561630"} Nov 03 21:52:15.916520 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.916498 5973 generic.go:356] "Generic (PLEG): container finished" podID="193d100c-0a0f-4568-9705-905ea314479e" containerID="8344c63b983309aef3001b1927fb3ac5a5b8eec32b516f8756db52adee662c00" exitCode=0 Nov 03 21:52:15.916619 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.916575 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" event={"ID":"193d100c-0a0f-4568-9705-905ea314479e","Type":"ContainerDied","Data":"8344c63b983309aef3001b1927fb3ac5a5b8eec32b516f8756db52adee662c00"} Nov 03 21:52:15.918600 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.918583 5973 generic.go:356] "Generic (PLEG): container finished" podID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerID="e3784c380746cf754471d913a2901a59c115b83bdd2638a14acef22046f89b20" exitCode=0 Nov 03 21:52:15.918698 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:15.918669 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" event={"ID":"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe","Type":"ContainerDied","Data":"e3784c380746cf754471d913a2901a59c115b83bdd2638a14acef22046f89b20"} Nov 03 21:52:17.054148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.054121 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:17.082586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.082559 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\") pod \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " Nov 03 21:52:17.082705 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.082644 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\") pod \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " Nov 03 21:52:17.082790 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.082767 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\" (UniqueName: \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\") pod \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\" (UID: \"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb\") " Nov 03 21:52:17.084531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.084502 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle" pod "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" (UID: "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.085899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.085859 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls" pod "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" (UID: "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb"). InnerVolumeSpecName "kube-api-access-qvsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:17.092800 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.092749 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util" pod "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" (UID: "4248bdd8-6e9f-4595-bf1a-6bf82945c3eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.100999 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.100966 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:17.134371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.134348 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:17.137364 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.137348 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:17.184320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184255 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\") pod \"11ffa7f1-5a95-4240-b6c3-09d434bde701\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " Nov 03 21:52:17.184320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184308 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\") pod \"193d100c-0a0f-4568-9705-905ea314479e\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " Nov 03 21:52:17.184545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184371 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\" (UniqueName: \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\") pod \"11ffa7f1-5a95-4240-b6c3-09d434bde701\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " Nov 03 21:52:17.184545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184403 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\" (UniqueName: \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\") pod \"193d100c-0a0f-4568-9705-905ea314479e\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " Nov 03 21:52:17.184545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184446 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\") pod \"11ffa7f1-5a95-4240-b6c3-09d434bde701\" (UID: \"11ffa7f1-5a95-4240-b6c3-09d434bde701\") " Nov 03 21:52:17.184545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184501 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\") pod \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " Nov 03 21:52:17.184759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184566 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\") pod \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " Nov 03 21:52:17.184759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184597 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\") pod \"193d100c-0a0f-4568-9705-905ea314479e\" (UID: \"193d100c-0a0f-4568-9705-905ea314479e\") " Nov 03 21:52:17.184759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184651 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\" (UniqueName: \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\") pod \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\" (UID: \"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe\") " Nov 03 21:52:17.185013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184927 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.185013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184821 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle" pod "11ffa7f1-5a95-4240-b6c3-09d434bde701" (UID: "11ffa7f1-5a95-4240-b6c3-09d434bde701"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.185013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184954 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvsls\" (UniqueName: \"kubernetes.io/projected/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-kube-api-access-qvsls\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.185013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.184970 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4248bdd8-6e9f-4595-bf1a-6bf82945c3eb-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.185487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.185423 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle" pod "193d100c-0a0f-4568-9705-905ea314479e" (UID: "193d100c-0a0f-4568-9705-905ea314479e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.185952 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.185927 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle" pod "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" (UID: "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.187182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.187139 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9" pod "11ffa7f1-5a95-4240-b6c3-09d434bde701" (UID: "11ffa7f1-5a95-4240-b6c3-09d434bde701"). InnerVolumeSpecName "kube-api-access-2zsd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:17.187639 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.187614 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p" pod "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" (UID: "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe"). InnerVolumeSpecName "kube-api-access-s2w8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:17.188554 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.188533 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68" pod "193d100c-0a0f-4568-9705-905ea314479e" (UID: "193d100c-0a0f-4568-9705-905ea314479e"). InnerVolumeSpecName "kube-api-access-mvl68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:17.191719 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.191693 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util" pod "11ffa7f1-5a95-4240-b6c3-09d434bde701" (UID: "11ffa7f1-5a95-4240-b6c3-09d434bde701"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.193882 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.193859 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util" pod "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" (UID: "154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.194233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.194213 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util" pod "193d100c-0a0f-4568-9705-905ea314479e" (UID: "193d100c-0a0f-4568-9705-905ea314479e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:52:17.285350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285323 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285345 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285355 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zsd9\" (UniqueName: \"kubernetes.io/projected/11ffa7f1-5a95-4240-b6c3-09d434bde701-kube-api-access-2zsd9\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285364 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl68\" (UniqueName: \"kubernetes.io/projected/193d100c-0a0f-4568-9705-905ea314479e-kube-api-access-mvl68\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285375 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11ffa7f1-5a95-4240-b6c3-09d434bde701-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285384 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285393 5973 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-bundle\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285400 5973 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/193d100c-0a0f-4568-9705-905ea314479e-util\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.285547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.285407 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2w8p\" (UniqueName: \"kubernetes.io/projected/154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe-kube-api-access-s2w8p\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:52:17.929803 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.929764 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" event={"ID":"4248bdd8-6e9f-4595-bf1a-6bf82945c3eb","Type":"ContainerDied","Data":"bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b"} Nov 03 21:52:17.929803 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.929793 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdpkz" Nov 03 21:52:17.930034 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.929820 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbae69d3b8f699555bac8cc2c13ae2e945772b7ae05f353814aeb829ca4cca8b" Nov 03 21:52:17.931882 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.931864 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" Nov 03 21:52:17.931882 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.931876 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bslkx" event={"ID":"11ffa7f1-5a95-4240-b6c3-09d434bde701","Type":"ContainerDied","Data":"927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398"} Nov 03 21:52:17.932070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.931909 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927387a566312b9c037a9fd33183a2933ce569852d5603e07f1930e66c138398" Nov 03 21:52:17.933912 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.933879 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" event={"ID":"193d100c-0a0f-4568-9705-905ea314479e","Type":"ContainerDied","Data":"4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1"} Nov 03 21:52:17.933912 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.933895 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amn65g" Nov 03 21:52:17.934079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.933917 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7155308425d03e5307751ddb4fb10c121fd706b840e62cec7e0d6c935e91e1" Nov 03 21:52:17.935872 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.935854 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" Nov 03 21:52:17.935872 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.935864 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jzb4" event={"ID":"154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe","Type":"ContainerDied","Data":"357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487"} Nov 03 21:52:17.936018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:17.935901 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357070b9567f576e0c1b147a6b9af9e1fba56d1253f5dba7136ce036187ed487" Nov 03 21:52:19.955635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:19.955597 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:20.759118 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:20.759081 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:23.820421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:23.820376 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:24.002886 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:24.002853 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:24.166606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:24.166570 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:24.791294 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:24.791260 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:26.418388 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:26.418353 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:28.167226 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:28.167195 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:28.491376 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:28.491285 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:30.428410 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:30.428359 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:32.193338 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:32.193302 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:33.175624 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:33.175586 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:33.335679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:33.335645 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:33.567739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:33.567651 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:33.723694 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:33.723660 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:36.285725 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.285695 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28"] Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286003 5973 cpu_manager.go:414] "containerMap: removing container" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="util" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286012 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="util" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286021 5973 cpu_manager.go:414] "containerMap: removing container" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="extract" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286026 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="extract" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286041 5973 cpu_manager.go:414] "containerMap: removing container" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="pull" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286047 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="pull" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286053 5973 cpu_manager.go:414] "containerMap: removing container" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="extract" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286057 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="extract" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286070 5973 cpu_manager.go:414] "containerMap: removing container" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="util" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286075 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="util" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286082 5973 cpu_manager.go:414] "containerMap: removing container" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="extract" Nov 03 21:52:36.286085 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286087 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286095 5973 cpu_manager.go:414] "containerMap: removing container" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286099 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286106 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="util" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286110 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="util" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286118 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286122 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286128 5973 cpu_manager.go:414] "containerMap: removing container" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="util" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286133 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="util" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286138 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286142 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286150 5973 cpu_manager.go:414] "containerMap: removing container" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286154 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="pull" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286198 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="4248bdd8-6e9f-4595-bf1a-6bf82945c3eb" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286206 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="193d100c-0a0f-4568-9705-905ea314479e" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286214 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="154b97d3-8aa4-4c4e-ab1f-ff1fa2309ebe" containerName="extract" Nov 03 21:52:36.286436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.286221 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ffa7f1-5a95-4240-b6c3-09d434bde701" containerName="extract" Nov 03 21:52:36.290864 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.290845 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.298079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.298041 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28"] Nov 03 21:52:36.346837 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.346803 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blq65\" (UniqueName: \"kubernetes.io/projected/fb871a0f-2db0-483b-af0f-818f79cd9b31-kube-api-access-blq65\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.347003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.346849 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.347003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.346941 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb871a0f-2db0-483b-af0f-818f79cd9b31-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.369389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.369362 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 21:52:36.369614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.369596 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l7fpp\"" Nov 03 21:52:36.371159 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.371125 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 21:52:36.371356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.371331 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 21:52:36.371356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.371337 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 21:52:36.371629 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.371588 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:36.448098 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.448062 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb871a0f-2db0-483b-af0f-818f79cd9b31-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.448225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.448132 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blq65\" (UniqueName: \"kubernetes.io/projected/fb871a0f-2db0-483b-af0f-818f79cd9b31-kube-api-access-blq65\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.448225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.448167 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.448322 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:52:36.448305 5973 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 21:52:36.448402 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:52:36.448390 5973 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert podName:fb871a0f-2db0-483b-af0f-818f79cd9b31 nodeName:}" failed. No retries permitted until 2026-11-03 21:52:36.948368312 +0000 UTC m=+396.723218178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-9qw28" (UID: "fb871a0f-2db0-483b-af0f-818f79cd9b31") : secret "plugin-serving-cert" not found Nov 03 21:52:36.448789 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.448761 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb871a0f-2db0-483b-af0f-818f79cd9b31-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.462093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.462068 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blq65\" (UniqueName: \"kubernetes.io/projected/fb871a0f-2db0-483b-af0f-818f79cd9b31-kube-api-access-blq65\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.579922 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.579832 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:36.952927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.952889 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:36.955427 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:36.955401 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb871a0f-2db0-483b-af0f-818f79cd9b31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9qw28\" (UID: \"fb871a0f-2db0-483b-af0f-818f79cd9b31\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:37.203288 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:37.203214 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" Nov 03 21:52:37.475786 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:37.475758 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28"] Nov 03 21:52:37.477990 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:52:37.477966 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb871a0f_2db0_483b_af0f_818f79cd9b31.slice/crio-f20c9189bd330d1a580c94c62ea6c3a804d24b8d35f9598c7dd05d964227c95a WatchSource:0}: Error finding container f20c9189bd330d1a580c94c62ea6c3a804d24b8d35f9598c7dd05d964227c95a: Status 404 returned error can't find the container with id f20c9189bd330d1a580c94c62ea6c3a804d24b8d35f9598c7dd05d964227c95a Nov 03 21:52:37.970900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:37.970862 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:38.024602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:38.024565 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" event={"ID":"fb871a0f-2db0-483b-af0f-818f79cd9b31","Type":"ContainerStarted","Data":"f20c9189bd330d1a580c94c62ea6c3a804d24b8d35f9598c7dd05d964227c95a"} Nov 03 21:52:38.124987 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:38.124957 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:38.380171 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:38.380100 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:38.534938 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:38.534892 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:38.693840 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:38.693803 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:43.054143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:43.054108 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" event={"ID":"fb871a0f-2db0-483b-af0f-818f79cd9b31","Type":"ContainerStarted","Data":"731da1b0cb2cbf07cf28e25e3589f262c1c27c624215d73dd3b763ea1f279b7e"} Nov 03 21:52:43.071576 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:43.071508 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9qw28" podStartSLOduration=1.936087363 podStartE2EDuration="7.071493385s" totalImagesPullingTime="5.135406022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:43.070904224 +0000 UTC m=+402.845754071" watchObservedRunningTime="2026-11-03 21:52:43.071493385 +0000 UTC m=+402.846343235" Nov 03 21:52:44.273044 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:44.273011 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:44.398184 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:52:44.398153 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:52:44.434866 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:44.434832 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:49.031294 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:49.031252 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:49.187073 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:49.187033 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.810929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:52.810891 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.964398 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:52.964349 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:53.117229 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:53.117115 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:55.994075 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:55.994045 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:56.942470 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:56.942408 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:57.091130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:52:57.091097 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:03.440384 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:03.440337 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:03.594164 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:03.594119 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:08.179755 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:08.179716 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:08.332790 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:08.332756 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:08.571768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:08.571696 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:08.727388 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:08.727354 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:11.230686 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:11.230655 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:11.383097 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:11.383060 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:12.186192 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:12.186158 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:12.341348 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:12.341313 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:14.300862 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:14.300827 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:14.398887 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:53:14.398851 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:53:16.985428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:16.985390 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.140625 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:17.140593 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.401725 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:17.401690 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.557018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:17.556967 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.788296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:17.788207 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.938696 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:17.938662 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:18.298403 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:18.298344 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:18.460003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:18.459968 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:23.145803 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:23.145752 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:23.293270 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:23.293224 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:27.883841 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:27.883801 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:28.042270 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:28.042239 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:32.534621 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.534583 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:32.685029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.684987 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:32.884752 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.884681 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:53:32.888294 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.888274 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:32.899116 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.899090 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:53:32.920425 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.920397 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 21:53:32.934440 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.934411 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce277295-b834-4149-9c99-e921a9277b32-config-file\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:32.934627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.934532 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvxv\" (UniqueName: \"kubernetes.io/projected/ce277295-b834-4149-9c99-e921a9277b32-kube-api-access-nxvxv\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:32.935003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:32.934983 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:53:33.035757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.035720 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce277295-b834-4149-9c99-e921a9277b32-config-file\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:33.035944 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.035788 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvxv\" (UniqueName: \"kubernetes.io/projected/ce277295-b834-4149-9c99-e921a9277b32-kube-api-access-nxvxv\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:33.036353 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.036331 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce277295-b834-4149-9c99-e921a9277b32-config-file\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:33.053374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.053343 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvxv\" (UniqueName: \"kubernetes.io/projected/ce277295-b834-4149-9c99-e921a9277b32-kube-api-access-nxvxv\") pod \"limitador-limitador-7c9b87d574-qq89p\" (UID: \"ce277295-b834-4149-9c99-e921a9277b32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:33.201572 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.201536 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:33.300808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.300770 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:33.304386 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.304365 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:33.313172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.313151 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nqlk2\"" Nov 03 21:53:33.314598 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.314575 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:33.337615 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.337583 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs6s\" (UniqueName: \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\") pod \"authorino-57f7f8659f-bh5v8\" (UID: \"5132aa21-5794-4861-bfe1-9755ccdf25d8\") " pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:33.410426 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.410386 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:33.438622 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.438577 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs6s\" (UniqueName: \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\") pod \"authorino-57f7f8659f-bh5v8\" (UID: \"5132aa21-5794-4861-bfe1-9755ccdf25d8\") " pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:33.452271 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.452194 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs6s\" (UniqueName: \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\") pod \"authorino-57f7f8659f-bh5v8\" (UID: \"5132aa21-5794-4861-bfe1-9755ccdf25d8\") " pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:33.477286 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.477205 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:53:33.479712 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:53:33.479677 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce277295_b834_4149_9c99_e921a9277b32.slice/crio-58f9ba353655f2b83b4ce98f7b1f806c10e8c6aa0190f1ee412d4b2eee0dc8c1 WatchSource:0}: Error finding container 58f9ba353655f2b83b4ce98f7b1f806c10e8c6aa0190f1ee412d4b2eee0dc8c1: Status 404 returned error can't find the container with id 58f9ba353655f2b83b4ce98f7b1f806c10e8c6aa0190f1ee412d4b2eee0dc8c1 Nov 03 21:53:33.616573 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.616521 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:33.626621 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.626595 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:33.890673 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:33.890648 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:33.892937 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:53:33.892911 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5132aa21_5794_4861_bfe1_9755ccdf25d8.slice/crio-270945db6b28853f71d57c215a8131c92d07031eb04275d6d20faf18677a495c WatchSource:0}: Error finding container 270945db6b28853f71d57c215a8131c92d07031eb04275d6d20faf18677a495c: Status 404 returned error can't find the container with id 270945db6b28853f71d57c215a8131c92d07031eb04275d6d20faf18677a495c Nov 03 21:53:34.306895 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:34.306856 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" event={"ID":"5132aa21-5794-4861-bfe1-9755ccdf25d8","Type":"ContainerStarted","Data":"270945db6b28853f71d57c215a8131c92d07031eb04275d6d20faf18677a495c"} Nov 03 21:53:34.308080 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:34.308055 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" event={"ID":"ce277295-b834-4149-9c99-e921a9277b32","Type":"ContainerStarted","Data":"58f9ba353655f2b83b4ce98f7b1f806c10e8c6aa0190f1ee412d4b2eee0dc8c1"} Nov 03 21:53:35.510999 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:35.510944 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:36.267473 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:36.267381 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:37.330828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.330779 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" event={"ID":"5132aa21-5794-4861-bfe1-9755ccdf25d8","Type":"ContainerStarted","Data":"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a"} Nov 03 21:53:37.331357 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.330846 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" podUID="5132aa21-5794-4861-bfe1-9755ccdf25d8" containerName="authorino" containerID="cri-o://35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a" gracePeriod=30 Nov 03 21:53:37.333256 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.333226 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" event={"ID":"ce277295-b834-4149-9c99-e921a9277b32","Type":"ContainerStarted","Data":"526f12053d3ec504bceecd688823d3bc486a0a2d1b5520d0fbb3baa16928b814"} Nov 03 21:53:37.333555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.333537 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:37.349069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.349009 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" podStartSLOduration=1.03788341 podStartE2EDuration="4.348995878s" totalImagesPullingTime="3.311112468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:37.347176497 +0000 UTC m=+457.122026346" watchObservedRunningTime="2026-11-03 21:53:37.348995878 +0000 UTC m=+457.123845751" Nov 03 21:53:37.367127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.367079 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" podStartSLOduration=2.5877880859999998 podStartE2EDuration="5.367064368s" totalImagesPullingTime="2.779276282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:37.366017105 +0000 UTC m=+457.140866955" watchObservedRunningTime="2026-11-03 21:53:37.367064368 +0000 UTC m=+457.141914214" Nov 03 21:53:37.579391 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.579365 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:37.679931 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.679883 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\" (UniqueName: \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\") pod \"5132aa21-5794-4861-bfe1-9755ccdf25d8\" (UID: \"5132aa21-5794-4861-bfe1-9755ccdf25d8\") " Nov 03 21:53:37.682313 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.682289 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s" pod "5132aa21-5794-4861-bfe1-9755ccdf25d8" (UID: "5132aa21-5794-4861-bfe1-9755ccdf25d8"). InnerVolumeSpecName "kube-api-access-zfs6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:37.781054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:37.780955 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfs6s\" (UniqueName: \"kubernetes.io/projected/5132aa21-5794-4861-bfe1-9755ccdf25d8-kube-api-access-zfs6s\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:53:38.339331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.339295 5973 generic.go:356] "Generic (PLEG): container finished" podID="5132aa21-5794-4861-bfe1-9755ccdf25d8" containerID="35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a" exitCode=0 Nov 03 21:53:38.339792 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.339406 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" Nov 03 21:53:38.343966 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.343940 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" event={"ID":"5132aa21-5794-4861-bfe1-9755ccdf25d8","Type":"ContainerDied","Data":"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a"} Nov 03 21:53:38.344104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.343973 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bh5v8" event={"ID":"5132aa21-5794-4861-bfe1-9755ccdf25d8","Type":"ContainerDied","Data":"270945db6b28853f71d57c215a8131c92d07031eb04275d6d20faf18677a495c"} Nov 03 21:53:38.344104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.343991 5973 scope.go:122] "RemoveContainer" containerID="35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a" Nov 03 21:53:38.356119 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.356097 5973 scope.go:122] "RemoveContainer" containerID="35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a" Nov 03 21:53:38.356604 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:53:38.356580 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a\": container with ID starting with 35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a not found: ID does not exist" containerID="35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a" Nov 03 21:53:38.356721 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.356612 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a"} err="failed to get container status \"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a\": rpc error: code = NotFound desc = could not find container \"35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a\": container with ID starting with 35f2e272fdfcb6d112262621f90adaa275da295bdefa58f842f6fef18f69dc9a not found: ID does not exist" Nov 03 21:53:38.366516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.366490 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:38.368278 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:38.368255 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bh5v8"] Nov 03 21:53:40.344296 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:40.344223 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5132aa21-5794-4861-bfe1-9755ccdf25d8" path="/var/lib/kubelet/pods/5132aa21-5794-4861-bfe1-9755ccdf25d8/volumes" Nov 03 21:53:44.399292 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:53:44.399252 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:53:48.343922 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:48.343893 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qq89p" Nov 03 21:53:52.836242 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:52.836206 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:52.995785 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:52.995750 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:53.608476 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:53.608428 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:56.053992 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:56.053955 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:57.707786 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:53:57.707746 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:06.111232 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.111188 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:06.269665 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.269625 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:06.343340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.343314 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pzm84"] Nov 03 21:54:06.343656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.343644 5973 cpu_manager.go:414] "containerMap: removing container" podUID="5132aa21-5794-4861-bfe1-9755ccdf25d8" containerName="authorino" Nov 03 21:54:06.343703 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.343658 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5132aa21-5794-4861-bfe1-9755ccdf25d8" containerName="authorino" Nov 03 21:54:06.343737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.343719 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="5132aa21-5794-4861-bfe1-9755ccdf25d8" containerName="authorino" Nov 03 21:54:06.347963 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.347948 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.350107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.350084 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pzm84"] Nov 03 21:54:06.354202 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.354162 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nqlk2\"" Nov 03 21:54:06.431194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.431176 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq8v\" (UniqueName: \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\") pod \"authorino-6f6944bcdd-pzm84\" (UID: \"d63ab70e-bc34-4ff4-b729-36f283ccfa9e\") " pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.438731 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.438701 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pzm84"] Nov 03 21:54:06.438874 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:54:06.438858 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gpq8v], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-pzm84" podUID="d63ab70e-bc34-4ff4-b729-36f283ccfa9e" Nov 03 21:54:06.465693 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.465669 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.472356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.472335 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.532577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.532545 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq8v\" (UniqueName: \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\") pod \"authorino-6f6944bcdd-pzm84\" (UID: \"d63ab70e-bc34-4ff4-b729-36f283ccfa9e\") " pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.544818 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.544789 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq8v\" (UniqueName: \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\") pod \"authorino-6f6944bcdd-pzm84\" (UID: \"d63ab70e-bc34-4ff4-b729-36f283ccfa9e\") " pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:06.633362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.633336 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\" (UniqueName: \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\") pod \"d63ab70e-bc34-4ff4-b729-36f283ccfa9e\" (UID: \"d63ab70e-bc34-4ff4-b729-36f283ccfa9e\") " Nov 03 21:54:06.635435 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.635411 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v" pod "d63ab70e-bc34-4ff4-b729-36f283ccfa9e" (UID: "d63ab70e-bc34-4ff4-b729-36f283ccfa9e"). InnerVolumeSpecName "kube-api-access-gpq8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:06.717805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.717745 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:54:06.719951 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.719934 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.723452 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.723435 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 21:54:06.729123 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.729083 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:54:06.734608 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.734589 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpq8v\" (UniqueName: \"kubernetes.io/projected/d63ab70e-bc34-4ff4-b729-36f283ccfa9e-kube-api-access-gpq8v\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:06.835365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.835342 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.835526 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.835380 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5cc\" (UniqueName: \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.936087 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.936059 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.936187 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.936099 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5cc\" (UniqueName: \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.938662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.938635 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:06.948675 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:06.948654 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5cc\" (UniqueName: \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\") pod \"authorino-6ffc44d9b-42r42\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:07.030619 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.030563 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:54:07.301617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.301591 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:54:07.303996 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:54:07.303975 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ab9a3b7_488e_465b_b14b_e20ae1f2048a.slice/crio-19c1da3668969aa3a885ce39a15d275d1b99b8cc0a13066560878a7ca7fb5bbb WatchSource:0}: Error finding container 19c1da3668969aa3a885ce39a15d275d1b99b8cc0a13066560878a7ca7fb5bbb: Status 404 returned error can't find the container with id 19c1da3668969aa3a885ce39a15d275d1b99b8cc0a13066560878a7ca7fb5bbb Nov 03 21:54:07.471658 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.471634 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pzm84" Nov 03 21:54:07.471658 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.471636 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffc44d9b-42r42" event={"ID":"2ab9a3b7-488e-465b-b14b-e20ae1f2048a","Type":"ContainerStarted","Data":"19c1da3668969aa3a885ce39a15d275d1b99b8cc0a13066560878a7ca7fb5bbb"} Nov 03 21:54:07.502740 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.502715 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pzm84"] Nov 03 21:54:07.509148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:07.509126 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pzm84"] Nov 03 21:54:08.351271 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:08.351241 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63ab70e-bc34-4ff4-b729-36f283ccfa9e" path="/var/lib/kubelet/pods/d63ab70e-bc34-4ff4-b729-36f283ccfa9e/volumes" Nov 03 21:54:08.477751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:08.477717 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffc44d9b-42r42" event={"ID":"2ab9a3b7-488e-465b-b14b-e20ae1f2048a","Type":"ContainerStarted","Data":"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37"} Nov 03 21:54:08.498170 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:08.498090 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6ffc44d9b-42r42" podStartSLOduration=1.566284117 podStartE2EDuration="2.49807766s" totalImagesPullingTime="931.793543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:08.496423136 +0000 UTC m=+488.271272986" watchObservedRunningTime="2026-11-03 21:54:08.49807766 +0000 UTC m=+488.272927508" Nov 03 21:54:14.399050 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:54:14.399008 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:54:15.639810 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:15.639776 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.795411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:15.795370 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.945178 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:15.945140 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:19.571786 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:19.571754 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:19.741071 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:19.741033 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:23.297335 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:23.297299 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:23.451634 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:23.451599 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:39.109617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:39.109547 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:41.443668 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:41.443617 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:44.398391 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:54:44.398350 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:54:51.708116 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:51.708079 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:51.874087 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:51.874052 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:52.833874 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:52.833842 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:52.988940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:52.988912 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:53.816864 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:53.816823 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:54:56.122467 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:54:56.122412 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:14.398041 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:55:14.397966 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:55:38.803351 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:38.803311 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:38.954418 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:38.954380 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:44.398249 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:55:44.398208 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:55:54.690349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:54.690317 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:54.846520 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:54.846485 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:56.176978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:56.176944 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:57.150892 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:57.150849 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:57.436037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:57.435962 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:55:59.497240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:59.497201 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:59.932742 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:59.932704 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:59.943043 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:55:59.943014 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:56:00.464292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.464261 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv"] Nov 03 21:56:00.467911 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.467885 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.477275 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.477245 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 21:56:00.479025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.479006 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lj52z\"" Nov 03 21:56:00.479279 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.479262 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 21:56:00.479496 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.479483 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 21:56:00.479752 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.479730 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv"] Nov 03 21:56:00.559013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.558971 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.559013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.559015 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/255a3973-a3ab-420c-adc2-6afa00b99c71-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.559579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.559038 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.559579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.559159 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.559579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.559191 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdjt\" (UniqueName: \"kubernetes.io/projected/255a3973-a3ab-420c-adc2-6afa00b99c71-kube-api-access-rqdjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.559579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.559271 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.566180 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.566150 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:00.660149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660115 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/255a3973-a3ab-420c-adc2-6afa00b99c71-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660161 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660189 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660217 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdjt\" (UniqueName: \"kubernetes.io/projected/255a3973-a3ab-420c-adc2-6afa00b99c71-kube-api-access-rqdjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660253 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660287 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660724 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660705 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.660779 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.660744 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.661118 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.661073 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.662768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.662747 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/255a3973-a3ab-420c-adc2-6afa00b99c71-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.662880 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.662866 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/255a3973-a3ab-420c-adc2-6afa00b99c71-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.672808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.672786 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdjt\" (UniqueName: \"kubernetes.io/projected/255a3973-a3ab-420c-adc2-6afa00b99c71-kube-api-access-rqdjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lbjgv\" (UID: \"255a3973-a3ab-420c-adc2-6afa00b99c71\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:00.781763 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:00.781679 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:01.056512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:01.056477 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv"] Nov 03 21:56:01.059493 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:01.059447 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255a3973_a3ab_420c_adc2_6afa00b99c71.slice/crio-5506e406315f212dfac32e5281116144d1b959683e8c5d93b76d64c11e495310 WatchSource:0}: Error finding container 5506e406315f212dfac32e5281116144d1b959683e8c5d93b76d64c11e495310: Status 404 returned error can't find the container with id 5506e406315f212dfac32e5281116144d1b959683e8c5d93b76d64c11e495310 Nov 03 21:56:01.429556 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:01.429520 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:02.018267 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.018230 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" event={"ID":"255a3973-a3ab-420c-adc2-6afa00b99c71","Type":"ContainerStarted","Data":"5506e406315f212dfac32e5281116144d1b959683e8c5d93b76d64c11e495310"} Nov 03 21:56:02.274003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.255110 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:56:02.819671 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.819613 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm"] Nov 03 21:56:02.825311 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.822901 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.827885 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.827853 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 21:56:02.849968 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.849931 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm"] Nov 03 21:56:02.939137 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.939109 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:02.981878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.981850 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a21998-3b3d-4421-a710-bbb050edb7f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.982012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.981882 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.982012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.981920 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ljj\" (UniqueName: \"kubernetes.io/projected/e2a21998-3b3d-4421-a710-bbb050edb7f3-kube-api-access-c7ljj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.982012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.981997 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.982130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.982017 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:02.982130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:02.982039 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.023795 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.023762 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" event={"ID":"255a3973-a3ab-420c-adc2-6afa00b99c71","Type":"ContainerStarted","Data":"5de346d18b50f1b5aae4bc7a0c5af06b7e64f9c27cf886a7397c850efda63a25"} Nov 03 21:56:03.045314 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.045268 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" podStartSLOduration=1.785114672 podStartE2EDuration="3.045255236s" totalImagesPullingTime="1.260140564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:03.043570356 +0000 UTC m=+602.818420205" watchObservedRunningTime="2026-11-03 21:56:03.045255236 +0000 UTC m=+602.820105083" Nov 03 21:56:03.083111 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083041 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a21998-3b3d-4421-a710-bbb050edb7f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083111 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083075 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083111 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083103 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ljj\" (UniqueName: \"kubernetes.io/projected/e2a21998-3b3d-4421-a710-bbb050edb7f3-kube-api-access-c7ljj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083134 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083150 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083173 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083541 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083554 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.083677 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.083661 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.085569 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.085549 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a21998-3b3d-4421-a710-bbb050edb7f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.085729 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.085711 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a21998-3b3d-4421-a710-bbb050edb7f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.095660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.095633 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ljj\" (UniqueName: \"kubernetes.io/projected/e2a21998-3b3d-4421-a710-bbb050edb7f3-kube-api-access-c7ljj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm\" (UID: \"e2a21998-3b3d-4421-a710-bbb050edb7f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.142639 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.142610 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:03.419492 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.419259 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm"] Nov 03 21:56:03.421836 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:03.421804 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a21998_3b3d_4421_a710_bbb050edb7f3.slice/crio-ace80ed8ff1978e9ab730624f19f090eb2f92320010515421270e445fe5a66a5 WatchSource:0}: Error finding container ace80ed8ff1978e9ab730624f19f090eb2f92320010515421270e445fe5a66a5: Status 404 returned error can't find the container with id ace80ed8ff1978e9ab730624f19f090eb2f92320010515421270e445fe5a66a5 Nov 03 21:56:03.782373 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.782347 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:03.871233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:03.871199 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.033950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:04.033874 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" event={"ID":"e2a21998-3b3d-4421-a710-bbb050edb7f3","Type":"ContainerStarted","Data":"66e3113e9133fc603e3d396a2beb5a92150893d41af38991d909f089cfd9acc7"} Nov 03 21:56:04.033950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:04.033923 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" event={"ID":"e2a21998-3b3d-4421-a710-bbb050edb7f3","Type":"ContainerStarted","Data":"ace80ed8ff1978e9ab730624f19f090eb2f92320010515421270e445fe5a66a5"} Nov 03 21:56:04.057588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:04.057530 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" podStartSLOduration=1.744015939 podStartE2EDuration="2.057514402s" totalImagesPullingTime="313.498463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:04.055023043 +0000 UTC m=+603.829872889" watchObservedRunningTime="2026-11-03 21:56:04.057514402 +0000 UTC m=+603.832364277" Nov 03 21:56:04.143695 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:04.143664 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:05.545916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:05.545885 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:56:06.278392 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.278362 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw"] Nov 03 21:56:06.281974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.281956 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.286380 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.286358 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 21:56:06.291833 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.291812 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw"] Nov 03 21:56:06.374728 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.374698 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:06.417737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.417710 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.417861 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.417828 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.417901 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.417858 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.417945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.417926 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47462ee-ac43-4f06-aeda-0dd21997a7be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.417984 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.417964 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/e47462ee-ac43-4f06-aeda-0dd21997a7be-kube-api-access-g7bc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.418017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.418007 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519026 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.518989 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519026 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519026 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519048 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47462ee-ac43-4f06-aeda-0dd21997a7be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519069 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/e47462ee-ac43-4f06-aeda-0dd21997a7be-kube-api-access-g7bc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519100 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519130 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519473 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519416 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519509 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.519600 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.519546 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.521277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.521259 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47462ee-ac43-4f06-aeda-0dd21997a7be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.521638 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.521623 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47462ee-ac43-4f06-aeda-0dd21997a7be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.531362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.531305 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/e47462ee-ac43-4f06-aeda-0dd21997a7be-kube-api-access-g7bc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw\" (UID: \"e47462ee-ac43-4f06-aeda-0dd21997a7be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.594013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.593989 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:06.871929 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:06.871900 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw"] Nov 03 21:56:06.873761 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:06.873730 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode47462ee_ac43_4f06_aeda_0dd21997a7be.slice/crio-1f57f94c4bb2e2ee645f9c4493d3df6359eae2aa45ff1b96413c939d0030462a WatchSource:0}: Error finding container 1f57f94c4bb2e2ee645f9c4493d3df6359eae2aa45ff1b96413c939d0030462a: Status 404 returned error can't find the container with id 1f57f94c4bb2e2ee645f9c4493d3df6359eae2aa45ff1b96413c939d0030462a Nov 03 21:56:07.050437 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:07.050402 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" event={"ID":"e47462ee-ac43-4f06-aeda-0dd21997a7be","Type":"ContainerStarted","Data":"1f57f94c4bb2e2ee645f9c4493d3df6359eae2aa45ff1b96413c939d0030462a"} Nov 03 21:56:08.057722 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:08.057680 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" event={"ID":"e47462ee-ac43-4f06-aeda-0dd21997a7be","Type":"ContainerStarted","Data":"fdb7b14eae02322e8809f247a5b055af9106a50e63e23c492f64929604e14430"} Nov 03 21:56:08.078836 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:08.078780 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podStartSLOduration=1.7739510520000001 podStartE2EDuration="2.07874877s" totalImagesPullingTime="304.797718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:08.077149406 +0000 UTC m=+607.851999254" watchObservedRunningTime="2026-11-03 21:56:08.07874877 +0000 UTC m=+607.853598618" Nov 03 21:56:08.594397 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:08.594362 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:09.656640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:09.656608 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:56:10.782550 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:10.782503 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:10.794752 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:10.794732 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:11.082202 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:11.082114 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lbjgv" Nov 03 21:56:13.143053 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:13.143023 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:13.155102 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:13.155074 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:14.096828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:14.096803 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm" Nov 03 21:56:14.398477 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:56:14.398422 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:56:16.594410 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:16.594374 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:16.606482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:16.606447 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:17.111527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:17.111498 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:56:19.471911 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.471870 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-564bd7d5c5-6g6pd"] Nov 03 21:56:19.475560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.475537 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.485123 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.485095 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564bd7d5c5-6g6pd"] Nov 03 21:56:19.540082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.540048 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1cd5aaf-0692-4257-8095-7882434c9927-tls-cert\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.540082 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.540080 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f2m\" (UniqueName: \"kubernetes.io/projected/a1cd5aaf-0692-4257-8095-7882434c9927-kube-api-access-w8f2m\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.640859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.640830 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1cd5aaf-0692-4257-8095-7882434c9927-tls-cert\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.640859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.640863 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f2m\" (UniqueName: \"kubernetes.io/projected/a1cd5aaf-0692-4257-8095-7882434c9927-kube-api-access-w8f2m\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.643430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.643404 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1cd5aaf-0692-4257-8095-7882434c9927-tls-cert\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.653600 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.653575 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f2m\" (UniqueName: \"kubernetes.io/projected/a1cd5aaf-0692-4257-8095-7882434c9927-kube-api-access-w8f2m\") pod \"authorino-564bd7d5c5-6g6pd\" (UID: \"a1cd5aaf-0692-4257-8095-7882434c9927\") " pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:19.788189 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:19.788081 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" Nov 03 21:56:20.060086 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:20.060064 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564bd7d5c5-6g6pd"] Nov 03 21:56:20.062124 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:20.062094 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1cd5aaf_0692_4257_8095_7882434c9927.slice/crio-b79f858e90737095bee11134c317a27d40e1eacfcb3865534181e429e7be86cc WatchSource:0}: Error finding container b79f858e90737095bee11134c317a27d40e1eacfcb3865534181e429e7be86cc: Status 404 returned error can't find the container with id b79f858e90737095bee11134c317a27d40e1eacfcb3865534181e429e7be86cc Nov 03 21:56:20.122664 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:20.122635 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" event={"ID":"a1cd5aaf-0692-4257-8095-7882434c9927","Type":"ContainerStarted","Data":"b79f858e90737095bee11134c317a27d40e1eacfcb3865534181e429e7be86cc"} Nov 03 21:56:22.134822 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.134732 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" event={"ID":"a1cd5aaf-0692-4257-8095-7882434c9927","Type":"ContainerStarted","Data":"98bf1e4a3851ef33160c8d634192fdd59b806b43efc8687ca1986f8ac44afd07"} Nov 03 21:56:22.182518 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.182440 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-564bd7d5c5-6g6pd" podStartSLOduration=1.453985696 podStartE2EDuration="3.182424646s" totalImagesPullingTime="1.72843895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:22.154106242 +0000 UTC m=+621.928956090" watchObservedRunningTime="2026-11-03 21:56:22.182424646 +0000 UTC m=+621.957274491" Nov 03 21:56:22.183769 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.183745 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:56:22.183964 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.183944 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6ffc44d9b-42r42" podUID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" containerName="authorino" containerID="cri-o://c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37" gracePeriod=30 Nov 03 21:56:22.432900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.432873 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:56:22.462786 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.462757 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\" (UniqueName: \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\") pod \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " Nov 03 21:56:22.481396 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.481361 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert" pod "2ab9a3b7-488e-465b-b14b-e20ae1f2048a" (UID: "2ab9a3b7-488e-465b-b14b-e20ae1f2048a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:22.563768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.563727 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\" (UniqueName: \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\") pod \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\" (UID: \"2ab9a3b7-488e-465b-b14b-e20ae1f2048a\") " Nov 03 21:56:22.564066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.564044 5973 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-tls-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:22.565986 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.565957 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc" pod "2ab9a3b7-488e-465b-b14b-e20ae1f2048a" (UID: "2ab9a3b7-488e-465b-b14b-e20ae1f2048a"). InnerVolumeSpecName "kube-api-access-5l5cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:22.665104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.665050 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5cc\" (UniqueName: \"kubernetes.io/projected/2ab9a3b7-488e-465b-b14b-e20ae1f2048a-kube-api-access-5l5cc\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:22.832680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.832646 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:22.992349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:22.992310 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:23.142032 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.142002 5973 generic.go:356] "Generic (PLEG): container finished" podID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" containerID="c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37" exitCode=0 Nov 03 21:56:23.142522 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.142047 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffc44d9b-42r42" Nov 03 21:56:23.142522 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.142081 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffc44d9b-42r42" event={"ID":"2ab9a3b7-488e-465b-b14b-e20ae1f2048a","Type":"ContainerDied","Data":"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37"} Nov 03 21:56:23.142522 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.142132 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffc44d9b-42r42" event={"ID":"2ab9a3b7-488e-465b-b14b-e20ae1f2048a","Type":"ContainerDied","Data":"19c1da3668969aa3a885ce39a15d275d1b99b8cc0a13066560878a7ca7fb5bbb"} Nov 03 21:56:23.142522 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.142154 5973 scope.go:122] "RemoveContainer" containerID="c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37" Nov 03 21:56:23.153400 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.153379 5973 scope.go:122] "RemoveContainer" containerID="c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37" Nov 03 21:56:23.153762 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:56:23.153736 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37\": container with ID starting with c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37 not found: ID does not exist" containerID="c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37" Nov 03 21:56:23.153827 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.153776 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37"} err="failed to get container status \"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37\": rpc error: code = NotFound desc = could not find container \"c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37\": container with ID starting with c484b50541890858a8b0b40b9bc00ff29b514c69a6d3d51e8a875e3c20ad9c37 not found: ID does not exist" Nov 03 21:56:23.167766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.167742 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:56:23.175597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:23.175483 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6ffc44d9b-42r42"] Nov 03 21:56:24.344527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:24.344494 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" path="/var/lib/kubelet/pods/2ab9a3b7-488e-465b-b14b-e20ae1f2048a/volumes" Nov 03 21:56:27.858670 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:27.858627 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:28.052852 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:28.052819 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:33.883650 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.883616 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb"] Nov 03 21:56:33.884171 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.884151 5973 cpu_manager.go:414] "containerMap: removing container" podUID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" containerName="authorino" Nov 03 21:56:33.884266 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.884175 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" containerName="authorino" Nov 03 21:56:33.884321 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.884269 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab9a3b7-488e-465b-b14b-e20ae1f2048a" containerName="authorino" Nov 03 21:56:33.890742 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.890719 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.895105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.895086 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tn75j\"" Nov 03 21:56:33.900323 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.900302 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb"] Nov 03 21:56:33.900504 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.900414 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 21:56:33.978141 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978111 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978299 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978150 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978299 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978167 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7bm\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-kube-api-access-gr7bm\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978387 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978311 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978387 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978338 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978405 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978443 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978499 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:33.978635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:33.978538 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.079719 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079693 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.079848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079731 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.079848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079748 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7bm\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-kube-api-access-gr7bm\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.079848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079808 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.079848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079844 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079897 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079927 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079952 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.079993 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080249 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.080225 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080366 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.080272 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080584 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.080558 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080703 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.080632 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.080828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.080803 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.082376 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.082351 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.082513 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.082495 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.092635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.092614 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7bm\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-kube-api-access-gr7bm\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.092716 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.092674 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/696e0ecd-a455-4af2-bdb4-9932fab6d32d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-5cffb\" (UID: \"696e0ecd-a455-4af2-bdb4-9932fab6d32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.206762 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.206738 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:34.487077 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.487050 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb"] Nov 03 21:56:34.489495 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:34.489468 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696e0ecd_a455_4af2_bdb4_9932fab6d32d.slice/crio-1655a1637e634021723e49a2da6a0525df7f88a00c8301c21fd3886340ba2a0c WatchSource:0}: Error finding container 1655a1637e634021723e49a2da6a0525df7f88a00c8301c21fd3886340ba2a0c: Status 404 returned error can't find the container with id 1655a1637e634021723e49a2da6a0525df7f88a00c8301c21fd3886340ba2a0c Nov 03 21:56:34.491421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:34.491404 5973 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:56:35.207678 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:35.207644 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" event={"ID":"696e0ecd-a455-4af2-bdb4-9932fab6d32d","Type":"ContainerStarted","Data":"1655a1637e634021723e49a2da6a0525df7f88a00c8301c21fd3886340ba2a0c"} Nov 03 21:56:37.526853 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:37.526820 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:37.527127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:37.526890 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:37.527127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:37.526916 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:38.224805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.224759 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" event={"ID":"696e0ecd-a455-4af2-bdb4-9932fab6d32d","Type":"ContainerStarted","Data":"0d6b031385f8fb608c9ebd015e7a9399a72a4b39d2c021380aa7f33677e23692"} Nov 03 21:56:38.250580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.250532 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podStartSLOduration=2.216298534 podStartE2EDuration="5.250519745s" totalImagesPullingTime="3.034221211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:38.248077575 +0000 UTC m=+638.022927422" watchObservedRunningTime="2026-11-03 21:56:38.250519745 +0000 UTC m=+638.025369657" Nov 03 21:56:38.439001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.438969 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f567997d9-rt9jg"] Nov 03 21:56:38.442541 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.442520 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.451000 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.450978 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 21:56:38.453183 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.453162 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f567997d9-rt9jg"] Nov 03 21:56:38.453281 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.453272 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 21:56:38.453481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.453430 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f8c6z\"" Nov 03 21:56:38.520760 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.520684 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgcw\" (UniqueName: \"kubernetes.io/projected/5bf69925-4d42-4335-b175-e9944add6bcb-kube-api-access-dfgcw\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.520900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.520798 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf69925-4d42-4335-b175-e9944add6bcb-metrics-certs\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.621576 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.621535 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf69925-4d42-4335-b175-e9944add6bcb-metrics-certs\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.622072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.621605 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgcw\" (UniqueName: \"kubernetes.io/projected/5bf69925-4d42-4335-b175-e9944add6bcb-kube-api-access-dfgcw\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.624195 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.624170 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf69925-4d42-4335-b175-e9944add6bcb-metrics-certs\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.638863 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.638838 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgcw\" (UniqueName: \"kubernetes.io/projected/5bf69925-4d42-4335-b175-e9944add6bcb-kube-api-access-dfgcw\") pod \"ai-gateway-controller-7f567997d9-rt9jg\" (UID: \"5bf69925-4d42-4335-b175-e9944add6bcb\") " pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:38.767666 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:38.767625 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:39.051634 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:39.051612 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f567997d9-rt9jg"] Nov 03 21:56:39.053836 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:39.053809 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf69925_4d42_4335_b175_e9944add6bcb.slice/crio-638dbcb7b418f8b2f83d178f384c1d1f54a2a88849930447fe017462828fca8c WatchSource:0}: Error finding container 638dbcb7b418f8b2f83d178f384c1d1f54a2a88849930447fe017462828fca8c: Status 404 returned error can't find the container with id 638dbcb7b418f8b2f83d178f384c1d1f54a2a88849930447fe017462828fca8c Nov 03 21:56:39.207273 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:39.207240 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:39.208490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:39.208444 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:39.208609 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:39.208513 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:39.230870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:39.230838 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" event={"ID":"5bf69925-4d42-4335-b175-e9944add6bcb","Type":"ContainerStarted","Data":"638dbcb7b418f8b2f83d178f384c1d1f54a2a88849930447fe017462828fca8c"} Nov 03 21:56:40.207728 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:40.207698 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:40.208091 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:40.207747 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:41.207625 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:41.207588 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:41.207812 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:41.207655 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:42.207440 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:42.207405 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:42.207647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:42.207482 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:42.281430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:42.281340 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" event={"ID":"5bf69925-4d42-4335-b175-e9944add6bcb","Type":"ContainerStarted","Data":"c26a1329bd6f4b612ac94f3df636295eb8480763a08546f99326941379505cca"} Nov 03 21:56:42.282049 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:42.282016 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:42.302038 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:42.301995 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" podStartSLOduration=1.325859444 podStartE2EDuration="4.301983868s" totalImagesPullingTime="2.976124424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:42.300269297 +0000 UTC m=+642.075119144" watchObservedRunningTime="2026-11-03 21:56:42.301983868 +0000 UTC m=+642.076833735" Nov 03 21:56:43.207281 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:43.207247 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:43.207451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:43.207295 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:44.207390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:44.207358 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 21:56:44.207774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:44.207411 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 21:56:44.399414 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:56:44.399375 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:56:45.556504 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:45.556441 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:45.556951 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:45.556738 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:45.557579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:45.557557 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" Nov 03 21:56:48.909685 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.909652 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj"] Nov 03 21:56:48.913377 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.913354 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:48.928733 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.928715 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj"] Nov 03 21:56:48.938982 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.938963 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp"] Nov 03 21:56:48.942262 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.942225 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:48.958340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:48.958313 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp"] Nov 03 21:56:49.040366 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040343 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040373 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040398 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040453 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b28807-4eee-4373-8d54-526f477ae078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040501 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040521 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040630 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040655 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsr7q\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-kube-api-access-hsr7q\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040673 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040786 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040773 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040822 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040795 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040853 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040824 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b28807-4eee-4373-8d54-526f477ae078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040874 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.040915 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040895 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjcc\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-kube-api-access-hrjcc\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040947 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040918 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3799a49d-c03b-43d4-b7ba-c2155e797733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.040993 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.040979 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.041025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.041014 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.041056 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.041034 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.141971 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142005 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsr7q\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-kube-api-access-hsr7q\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142033 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142080 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142104 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142140 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b28807-4eee-4373-8d54-526f477ae078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142192 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142238 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjcc\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-kube-api-access-hrjcc\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142270 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3799a49d-c03b-43d4-b7ba-c2155e797733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142300 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142330 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142359 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142400 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142429 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142498 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142523 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b28807-4eee-4373-8d54-526f477ae078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142549 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142580 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142584 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142645 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.142771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142668 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.143149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142913 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.143149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.142968 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.143149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.143105 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b28807-4eee-4373-8d54-526f477ae078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.143287 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.143161 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.143353 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.143331 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.143509 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.143484 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3799a49d-c03b-43d4-b7ba-c2155e797733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.143596 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.143531 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.145092 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.145071 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.145280 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.145259 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b28807-4eee-4373-8d54-526f477ae078-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.145344 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.145272 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.145521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.145504 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b28807-4eee-4373-8d54-526f477ae078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.155336 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.155312 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.155425 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.155407 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsr7q\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-kube-api-access-hsr7q\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.155791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.155775 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b28807-4eee-4373-8d54-526f477ae078-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-bkhgj\" (UID: \"b5b28807-4eee-4373-8d54-526f477ae078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.155941 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.155920 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjcc\" (UniqueName: \"kubernetes.io/projected/3799a49d-c03b-43d4-b7ba-c2155e797733-kube-api-access-hrjcc\") pod \"maas-default-gateway-openshift-default-66db998887-7z9gp\" (UID: \"3799a49d-c03b-43d4-b7ba-c2155e797733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.226384 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.226332 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:49.256992 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.256965 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:49.508500 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.508475 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj"] Nov 03 21:56:49.510638 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:49.510611 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b28807_4eee_4373_8d54_526f477ae078.slice/crio-b46061f54961f53e29e4bd4cbb42918cf2fc9d6efed6199a53aa009c5826c015 WatchSource:0}: Error finding container b46061f54961f53e29e4bd4cbb42918cf2fc9d6efed6199a53aa009c5826c015: Status 404 returned error can't find the container with id b46061f54961f53e29e4bd4cbb42918cf2fc9d6efed6199a53aa009c5826c015 Nov 03 21:56:49.513653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.513621 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:49.513739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.513686 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:49.513739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.513714 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:49.545369 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.545346 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp"] Nov 03 21:56:49.547889 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:56:49.547862 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3799a49d_c03b_43d4_b7ba_c2155e797733.slice/crio-ad63ff705b6a304dface068fc0b65f7b92a79d7b3fe5564625df9f90539dc4de WatchSource:0}: Error finding container ad63ff705b6a304dface068fc0b65f7b92a79d7b3fe5564625df9f90539dc4de: Status 404 returned error can't find the container with id ad63ff705b6a304dface068fc0b65f7b92a79d7b3fe5564625df9f90539dc4de Nov 03 21:56:49.554703 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.554673 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:49.554807 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.554738 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:49.554807 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:49.554767 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:56:50.330032 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.329996 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" event={"ID":"3799a49d-c03b-43d4-b7ba-c2155e797733","Type":"ContainerStarted","Data":"ca814fd3330a21456c511ce736be3475cfeb023d7b251ae0a79a10463766dc25"} Nov 03 21:56:50.330411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.330048 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" event={"ID":"3799a49d-c03b-43d4-b7ba-c2155e797733","Type":"ContainerStarted","Data":"ad63ff705b6a304dface068fc0b65f7b92a79d7b3fe5564625df9f90539dc4de"} Nov 03 21:56:50.331745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.331716 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" event={"ID":"b5b28807-4eee-4373-8d54-526f477ae078","Type":"ContainerStarted","Data":"d0bcffa07414df97f311c3c2aac3d831e151fe10b0b037c5416f43e472d19956"} Nov 03 21:56:50.331831 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.331816 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" event={"ID":"b5b28807-4eee-4373-8d54-526f477ae078","Type":"ContainerStarted","Data":"b46061f54961f53e29e4bd4cbb42918cf2fc9d6efed6199a53aa009c5826c015"} Nov 03 21:56:50.357686 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.356042 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podStartSLOduration=2.35602401 podStartE2EDuration="2.35602401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:50.354153473 +0000 UTC m=+650.129003321" watchObservedRunningTime="2026-11-03 21:56:50.35602401 +0000 UTC m=+650.130873858" Nov 03 21:56:50.408525 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:50.407192 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podStartSLOduration=2.407156141 podStartE2EDuration="2.407156141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:50.406018604 +0000 UTC m=+650.180868472" watchObservedRunningTime="2026-11-03 21:56:50.407156141 +0000 UTC m=+650.182005989" Nov 03 21:56:51.227419 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.227377 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:51.228961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.228934 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:51.229079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.228999 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:51.272482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.269753 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:51.276485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.272817 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:51.276485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:51.272884 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:52.226856 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:52.226811 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:52.227367 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:52.226884 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:52.259816 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:52.259784 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:52.260054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:52.260029 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:53.227628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:53.227595 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:53.228025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:53.227654 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:53.260702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:53.260670 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:53.260925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:53.260901 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:53.300483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:53.299698 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f567997d9-rt9jg" Nov 03 21:56:54.226900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:54.226872 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:54.227132 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:54.227109 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:54.263482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:54.259761 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:54.263482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:54.259819 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:55.055613 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.055575 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:55.226925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.226879 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:55.227125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.226940 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:55.271483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.260802 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:55.271483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.260859 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:55.561176 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:55.561106 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:56.227470 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:56.227428 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-bkhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 21:56:56.227643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:56.227524 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" podUID="b5b28807-4eee-4373-8d54-526f477ae078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 21:56:56.230219 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:56.230193 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:56.258360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:56.258327 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-7z9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 21:56:56.258560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:56.258383 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" podUID="3799a49d-c03b-43d4-b7ba-c2155e797733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 21:56:57.092107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.092041 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:57.246086 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.246047 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:57.802411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.802380 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:57.802845 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.802794 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:57.803831 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.803811 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-bkhgj" Nov 03 21:56:57.867615 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.867584 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:57.868413 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.868381 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:57.881590 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:57.878657 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-7z9gp" Nov 03 21:56:58.861446 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:58.861414 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:59.017436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:56:59.017406 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:14.399253 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:57:14.399209 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:57:20.847903 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:20.847872 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:20.999777 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:20.999749 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.218105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.218078 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:57:21.229610 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.229586 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.267529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.258741 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f4fb1b-openshift-default-dockercfg-pssfw\"" Nov 03 21:57:21.268555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.268530 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:57:21.377948 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.377916 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.377966 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378050 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378098 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378318 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378184 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378318 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378235 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378318 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378291 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378509 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378355 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b292s\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.378509 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.378385 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.424775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.424738 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 21:57:21.431231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.431204 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.442799 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.442769 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-53a2cb-openshift-default-dockercfg-6hq7h\"" Nov 03 21:57:21.445898 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.445875 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 21:57:21.447163 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.447136 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.479655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479596 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479625 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479645 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479660 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479783 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b292s\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479821 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.479955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479912 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.479974 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480009 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480050 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480106 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480398 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480372 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480473 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480407 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.480618 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.480492 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.482435 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.482414 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.482556 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.482517 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.492807 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.492783 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.492918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.492900 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b292s\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\") pod \"e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.529671 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.529643 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 21:57:21.534648 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.534627 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.543591 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.543560 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fc16de-openshift-default-dockercfg-thxpt\"" Nov 03 21:57:21.546361 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.546336 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 21:57:21.565487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.565451 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:21.581518 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581493 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581619 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581529 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581685 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581638 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581684 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581807 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581784 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581865 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581832 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.581924 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581907 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.582034 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581949 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.582034 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.581985 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwp6\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.627651 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.626353 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 21:57:21.627651 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.626970 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.632769 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.632631 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.647850 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.647638 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-02e8ab-openshift-default-dockercfg-4kgqm\"" Nov 03 21:57:21.649585 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.649559 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 21:57:21.682746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682714 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.682877 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682775 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.682877 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682853 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.682962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682886 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.682962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682916 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683030 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.682993 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683030 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683025 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683108 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683052 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683108 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683089 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683120 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683148 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683191 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683357 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683269 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683357 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683296 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683357 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683299 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683357 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683337 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwp6\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683575 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683477 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683575 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683518 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlprr\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683575 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683549 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683721 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683707 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.683774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683727 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.683955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683929 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.684074 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.683987 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.686573 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.686551 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.686804 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.686787 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.699197 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.699164 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.699197 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.699184 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwp6\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.704735 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.704712 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:57:21.710036 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.710021 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.727386 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.727351 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-85b4ff-openshift-default-dockercfg-m2k4x\"" Nov 03 21:57:21.728394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.728372 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:57:21.745589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.745530 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:21.785322 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785286 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.785507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785337 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.785507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785370 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlprr\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.785507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785407 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.785672 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785542 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.785672 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785582 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.785780 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785680 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.785780 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785725 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.785780 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785727 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.785953 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.785816 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.786139 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.786085 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.786711 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.786669 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.789721 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.789695 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.789816 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.789762 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.789881 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.789858 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.789945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.789893 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.789945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.789926 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.790058 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790042 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.790115 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790079 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.790169 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790113 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.790169 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790153 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.790272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790189 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5n2\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.790272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790211 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4qn\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.790272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790230 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.790272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790245 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.790272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790264 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.790614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790286 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.790614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790302 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.790614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790316 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.790614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790382 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.790826 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.790759 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.791107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.791090 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.792959 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.792936 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.793276 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.793247 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.804798 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.804771 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlprr\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.808259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.808034 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.813014 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.812988 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 21:57:21.819644 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.819628 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.820513 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.820443 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.833148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.833113 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 21:57:21.849676 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.849656 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:21.853274 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.853252 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:57:21.855788 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:21.855768 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5023934_6d80_4502_8b04_ee63751ed4f7.slice/crio-a151470e35c6adc0235f06965f5cb0fce6779c8f0a8dd92e865d6833b9a24326 WatchSource:0}: Error finding container a151470e35c6adc0235f06965f5cb0fce6779c8f0a8dd92e865d6833b9a24326: Status 404 returned error can't find the container with id a151470e35c6adc0235f06965f5cb0fce6779c8f0a8dd92e865d6833b9a24326 Nov 03 21:57:21.856153 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.856138 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8af3f3-openshift-default-dockercfg-hsqhd\"" Nov 03 21:57:21.859717 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.859687 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:21.859802 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.859763 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:21.859842 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.859806 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:21.892208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.892169 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.892365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.892226 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.892365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.892256 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.892365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.892294 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.892365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.892360 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.893041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893020 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.893183 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893111 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.893295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893249 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893290 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893372 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893490 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893528 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zq6\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893569 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893594 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893617 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893655 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893681 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.893968 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894007 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.894349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894205 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.895333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894406 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.895333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894493 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.895333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894561 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5n2\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.895537 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.894902 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4qn\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.895537 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.895408 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.895896 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.895876 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.896050 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.895941 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.896858 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.896832 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.896958 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.896587 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.896958 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.896932 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.897271 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.897223 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.897451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.897434 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.897701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.897686 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.897923 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.897907 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.898144 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.898127 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.898388 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.898373 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.898588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.898491 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.898774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.898726 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.901245 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.901220 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.901954 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.901448 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.902963 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.902944 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.913012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.912949 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.919066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.918958 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5n2\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\") pod \"e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.920222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.919843 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.927716 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.922276 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4qn\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\") pod \"e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:21.954647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.954614 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:21.984389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.984361 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:21.999761 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999679 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999761 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999734 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999776 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999813 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zq6\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999863 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999903 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999935 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:21.999962 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:21.999956 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.000287 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.000025 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.000830 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.000798 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.000967 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.000800 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.001046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.001024 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.001124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.001098 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.001585 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.001563 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.004660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.004636 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.005502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.005114 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.023135 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.022948 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:22.032654 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.032625 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zq6\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.039313 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.039288 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\") pod \"e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.039899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.039877 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 21:57:22.049087 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:22.049046 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc9e252_a5b5_45b1_87d2_0e102d35f61a.slice/crio-15f4d51cd439ef285d63cb49646fcb54ac11bfe00cc7d77805b047a26c20419a WatchSource:0}: Error finding container 15f4d51cd439ef285d63cb49646fcb54ac11bfe00cc7d77805b047a26c20419a: Status 404 returned error can't find the container with id 15f4d51cd439ef285d63cb49646fcb54ac11bfe00cc7d77805b047a26c20419a Nov 03 21:57:22.051747 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.051710 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.051848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.051795 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.051848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.051836 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.153855 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.153661 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:22.155990 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.155967 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.192310 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.192064 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 21:57:22.205928 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.205867 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.206086 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.206004 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.206166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.206059 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.255908 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.255885 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 21:57:22.270787 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.270750 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.270902 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.270838 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.270902 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.270883 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.333332 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.333235 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.367038 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.366923 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:57:22.381411 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:22.380220 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7290c8e9_990d_4032_96e3_501b933cdc9b.slice/crio-ede6724b0f6be08c28394b7237ad888a074863c9ba90aaa0ca493f1c134ac915 WatchSource:0}: Error finding container ede6724b0f6be08c28394b7237ad888a074863c9ba90aaa0ca493f1c134ac915: Status 404 returned error can't find the container with id ede6724b0f6be08c28394b7237ad888a074863c9ba90aaa0ca493f1c134ac915 Nov 03 21:57:22.384439 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.384363 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.384661 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.384624 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.385071 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.385036 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.475843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.475767 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 21:57:22.484279 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.481759 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.484279 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.481841 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.484279 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.481912 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:22.516245 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.516167 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerStarted","Data":"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b"} Nov 03 21:57:22.521697 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.521668 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerStarted","Data":"ede6724b0f6be08c28394b7237ad888a074863c9ba90aaa0ca493f1c134ac915"} Nov 03 21:57:22.522581 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.522560 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerStarted","Data":"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762"} Nov 03 21:57:22.523165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.523147 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerStarted","Data":"15f4d51cd439ef285d63cb49646fcb54ac11bfe00cc7d77805b047a26c20419a"} Nov 03 21:57:22.523295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.523280 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerStarted","Data":"d5e373253404c8ac32e1d43f0f33eaf7a3d3a48bf06750602582bb829029a424"} Nov 03 21:57:22.525130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.525102 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerStarted","Data":"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb"} Nov 03 21:57:22.525556 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.525531 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerStarted","Data":"599ca8faf34436f058e47a3aa4ba13ef007bea26877e8cafde18b1ad0bd5aaad"} Nov 03 21:57:22.527412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.527035 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerStarted","Data":"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62"} Nov 03 21:57:22.527412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.527068 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerStarted","Data":"f21caced290e170143fbc6a503e531d531b7127037573441853e6b0769abdcd4"} Nov 03 21:57:22.529766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.529231 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerStarted","Data":"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83"} Nov 03 21:57:22.529766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.529270 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerStarted","Data":"a151470e35c6adc0235f06965f5cb0fce6779c8f0a8dd92e865d6833b9a24326"} Nov 03 21:57:22.544213 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.542993 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podStartSLOduration=1.542977005 podStartE2EDuration="1.542977005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:22.542690804 +0000 UTC m=+682.317540657" watchObservedRunningTime="2026-11-03 21:57:22.542977005 +0000 UTC m=+682.317826858" Nov 03 21:57:22.566708 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.566569 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:22.570838 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.569632 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podStartSLOduration=1.569615508 podStartE2EDuration="1.569615508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:22.567944953 +0000 UTC m=+682.342794832" watchObservedRunningTime="2026-11-03 21:57:22.569615508 +0000 UTC m=+682.344465356" Nov 03 21:57:22.578918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.578893 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:22.621072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.621019 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podStartSLOduration=1.621001327 podStartE2EDuration="1.621001327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:22.593805913 +0000 UTC m=+682.368655761" watchObservedRunningTime="2026-11-03 21:57:22.621001327 +0000 UTC m=+682.395851175" Nov 03 21:57:22.622519 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.622478 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podStartSLOduration=1.622468537 podStartE2EDuration="1.622468537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:22.620380771 +0000 UTC m=+682.395230620" watchObservedRunningTime="2026-11-03 21:57:22.622468537 +0000 UTC m=+682.397318380" Nov 03 21:57:22.674064 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.674008 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podStartSLOduration=1.6739880249999999 podStartE2EDuration="1.673988025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:22.649854661 +0000 UTC m=+682.424704511" watchObservedRunningTime="2026-11-03 21:57:22.673988025 +0000 UTC m=+682.448837874" Nov 03 21:57:22.746037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.746005 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:22.850579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.850499 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:22.955407 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:22.955369 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:23.024054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.024024 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:23.029285 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.029261 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:23.535805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.535766 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerStarted","Data":"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13"} Nov 03 21:57:23.536443 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.536421 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:23.536719 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.536705 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:23.537426 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.537406 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:23.537597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.537584 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:23.562124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.562074 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podStartSLOduration=2.562057549 podStartE2EDuration="2.562057549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:23.560988769 +0000 UTC m=+683.335838617" watchObservedRunningTime="2026-11-03 21:57:23.562057549 +0000 UTC m=+683.336907397" Nov 03 21:57:23.752029 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.751996 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:23.856242 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.856168 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:23.960731 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:23.960703 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:24.154494 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.154445 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:24.159497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.159438 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:24.542305 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.542219 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:24.542305 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.542268 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:24.542305 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.542292 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:24.542587 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.542313 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:24.543163 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.543140 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:24.543305 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.543290 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:24.543347 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.543340 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:24.543391 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:24.543383 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:25.014243 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.014181 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:25.168896 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.168861 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:25.368346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.368261 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:25.560948 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.560913 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:25.720899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.720851 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:25.880985 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:25.880950 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:26.055795 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:26.055698 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:26.319978 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:26.319892 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:26.536240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:26.536192 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:27.175643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:27.175609 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:27.329691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:27.329654 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:27.959639 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:27.959591 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.127300 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:28.127267 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.288578 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:28.288491 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.596033 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:28.595950 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.744893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:28.744858 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.939966 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:28.939934 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.096718 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.096689 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.667491 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.667437 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.735621 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.735583 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:57:29.740179 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.740156 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:29.750037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.750011 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:57:29.807601 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.807571 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c0caef-dockercfg-lbw8x\"" Nov 03 21:57:29.810638 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.810618 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c0caef\"" Nov 03 21:57:29.828125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.828098 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.913334 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.913290 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:29.913691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.913663 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98cz\" (UniqueName: \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:29.980936 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:29.980840 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:30.015086 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.015056 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98cz\" (UniqueName: \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:30.015273 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.015214 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:30.015869 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.015849 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:30.027389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.027367 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98cz\" (UniqueName: \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\") pod \"payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:30.054869 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.054844 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:30.202140 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.202110 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:57:30.336452 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.336422 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:57:30.339090 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:30.339059 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddd4340_d564_4812_9596_92e770fa4534.slice/crio-5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57 WatchSource:0}: Error finding container 5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57: Status 404 returned error can't find the container with id 5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57 Nov 03 21:57:30.598324 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.598238 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" event={"ID":"addd4340-d564-4812-9596-92e770fa4534","Type":"ContainerStarted","Data":"5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57"} Nov 03 21:57:30.701122 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.701086 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:30.859563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:30.859482 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:31.009420 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:31.009367 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:31.711165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:31.711119 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:31.882933 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:31.882889 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:32.601483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:32.590595 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:32.781966 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:32.781881 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:33.407393 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:33.407359 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:33.579595 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:33.579255 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:33.622616 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:33.622585 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" event={"ID":"addd4340-d564-4812-9596-92e770fa4534","Type":"ContainerStarted","Data":"af74429d511493ed6f7a61d9e9504f7b07e649e96b475f37a7b9a0674eeb9f91"} Nov 03 21:57:33.622789 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:33.622702 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:33.642119 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:33.642073 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" podStartSLOduration=1.6772617749999998 podStartE2EDuration="4.642061274s" totalImagesPullingTime="2.964799499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:33.640354779 +0000 UTC m=+693.415204627" watchObservedRunningTime="2026-11-03 21:57:33.642061274 +0000 UTC m=+693.416911122" Nov 03 21:57:34.152629 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:34.152583 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:35.103782 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:35.103019 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:35.276481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:35.276100 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:35.853494 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:35.851854 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:36.556408 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:36.556372 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:37.548679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:37.548326 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:38.443231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:38.443182 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:39.077284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:39.077253 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:57:39.147638 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:39.145547 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:39.586479 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:39.585594 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:57:39.586479 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:39.585676 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:57:40.024157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:40.024065 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:40.192127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:40.191995 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:40.578070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:40.578032 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:40.781121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:40.781089 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:40.957886 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:40.957855 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:41.747950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:41.747917 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:42.521348 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:42.521181 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:43.039801 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.039687 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:57:43.044155 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.044126 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.050876 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-85b4ff\"/\"simulator-w5-85b4ff-kserve-self-signed-certs\"" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.051081 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-85b4ff\"/\"openshift-service-ca.crt\"" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.051147 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-85b4ff\"/\"default-dockercfg-2bw27\"" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.052590 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-85b4ff\"/\"kube-root-ca.crt\"" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053254 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053294 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9p5\" (UniqueName: \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053337 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053378 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053503 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.053702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.053529 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.063283 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.063203 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:57:43.135292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.135095 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163251 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163312 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9p5\" (UniqueName: \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163356 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163398 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163495 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.163519 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.178484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.176943 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.191484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.182207 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.209662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.209626 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.214953 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.214919 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.216513 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.216430 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.219521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.219494 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9p5\" (UniqueName: \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\") pod \"simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.419484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.419396 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:43.429326 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.429201 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:43.926580 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:43.926531 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:57:44.012875 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:44.012804 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" event={"ID":"8686da36-b720-489e-bbfa-d3d3e60ca278","Type":"ContainerStarted","Data":"9dc88c9873f683e8549f34d4dee13417ae0751e258dae9f5e0300faf013cf1cc"} Nov 03 21:57:44.195282 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:44.195026 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:44.442229 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:57:44.442143 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:57:44.631691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:44.630700 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:57:45.002774 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.000954 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:45.023390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.023352 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" event={"ID":"8686da36-b720-489e-bbfa-d3d3e60ca278","Type":"ContainerStarted","Data":"f7642d3a5eef585717418f0db55d2fe76c1f0d87607315f566bf49edbac18d7a"} Nov 03 21:57:45.044905 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.044795 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podStartSLOduration=2.044776547 podStartE2EDuration="2.044776547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:45.044342895 +0000 UTC m=+704.819192745" watchObservedRunningTime="2026-11-03 21:57:45.044776547 +0000 UTC m=+704.819626396" Nov 03 21:57:45.218544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.177934 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:45.430623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.430582 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:45.700784 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.700490 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:45.893221 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:45.893166 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:46.373971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:46.373921 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:57:46.941552 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:46.941446 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:47.534450 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.534414 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:57:47.583006 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.582912 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:57:47.583006 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.582997 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.589606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.588762 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f4fb1b\"/\"openshift-service-ca.crt\"" Nov 03 21:57:47.589606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.588953 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f4fb1b\"/\"kube-root-ca.crt\"" Nov 03 21:57:47.589606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.589046 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f4fb1b\"/\"simulator-w1-f4fb1b-kserve-self-signed-certs\"" Nov 03 21:57:47.589606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.589111 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f4fb1b\"/\"default-dockercfg-g9dgj\"" Nov 03 21:57:47.620077 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.619957 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:47.641016 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.640964 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.641234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.641026 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtc8w\" (UniqueName: \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.641234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.641067 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.641234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.641103 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.641234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.641188 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.641546 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.641319 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.743946 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.743899 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.744144 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.743969 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtc8w\" (UniqueName: \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.744745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.744720 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.744875 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.744766 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.744875 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.744845 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.745004 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.744983 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.745547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.745491 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.745698 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.745641 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.745770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.745707 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.747328 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.747301 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.748340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.748316 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.757791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.757729 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtc8w\" (UniqueName: \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\") pod \"simulator-w1-f4fb1b-kserve-99d675b87-c94h8\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:47.935107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:47.934778 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:49.048961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:48.058737 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:49.048961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:48.279503 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:57:49.048961 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:48.282599 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7630ca6a_44af_48f0_b81b_f2c671946c6e.slice/crio-f314444e970de26b7c4569bdc8facf3683e9757c56a61d7bab335f3df4315f98 WatchSource:0}: Error finding container f314444e970de26b7c4569bdc8facf3683e9757c56a61d7bab335f3df4315f98: Status 404 returned error can't find the container with id f314444e970de26b7c4569bdc8facf3683e9757c56a61d7bab335f3df4315f98 Nov 03 21:57:49.048961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:48.786733 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:49.085635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:49.085599 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" event={"ID":"7630ca6a-44af-48f0-b81b-f2c671946c6e","Type":"ContainerStarted","Data":"78187af9464932aa5d414006a7e33838828d5c7af911ad567e9ed165c3b52e14"} Nov 03 21:57:49.085808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:49.085646 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" event={"ID":"7630ca6a-44af-48f0-b81b-f2c671946c6e","Type":"ContainerStarted","Data":"f314444e970de26b7c4569bdc8facf3683e9757c56a61d7bab335f3df4315f98"} Nov 03 21:57:49.105785 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:49.105724 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" podStartSLOduration=2.10571111 podStartE2EDuration="2.10571111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:49.104508965 +0000 UTC m=+708.879358813" watchObservedRunningTime="2026-11-03 21:57:49.10571111 +0000 UTC m=+708.880560956" Nov 03 21:57:49.635883 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:49.635846 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:49.935709 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:49.935675 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:50.228343 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.228255 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:57:50.319808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.319777 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:50.332750 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.332721 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:57:50.332906 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.332827 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.339187 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.339150 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-02e8ab\"/\"kube-root-ca.crt\"" Nov 03 21:57:50.339345 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.339220 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-02e8ab\"/\"openshift-service-ca.crt\"" Nov 03 21:57:50.340214 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.340196 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-02e8ab\"/\"default-dockercfg-77r2d\"" Nov 03 21:57:50.340319 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.340214 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-02e8ab\"/\"simulator-w3-02e8ab-kserve-self-signed-certs\"" Nov 03 21:57:50.381033 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.380987 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.381240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.381042 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.381240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.381144 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.381240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.381223 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.381412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.381388 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6vf\" (UniqueName: \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.381504 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.381451 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.475659 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.475610 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:50.482650 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482564 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.482650 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482608 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.482650 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482645 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.482921 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482670 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.482921 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482713 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.482921 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.482857 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6vf\" (UniqueName: \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.483158 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.483127 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.483291 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.483195 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.483291 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.483257 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.485061 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.485040 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.485415 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.485393 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.495630 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.495596 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6vf\" (UniqueName: \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\") pod \"simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.645340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.645307 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:50.723934 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.723897 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:50.929385 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:50.929354 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.091548 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.087049 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.109470 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.109433 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:57:51.111922 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:57:51.111894 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6c8e21_db51_4cbf_81f2_868cbc1eff2f.slice/crio-a44e6bd3c0f167268c733780058ab309c88ba31ec8c6c97c95f47c636e19fe9b WatchSource:0}: Error finding container a44e6bd3c0f167268c733780058ab309c88ba31ec8c6c97c95f47c636e19fe9b: Status 404 returned error can't find the container with id a44e6bd3c0f167268c733780058ab309c88ba31ec8c6c97c95f47c636e19fe9b Nov 03 21:57:51.332795 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.332765 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.519723 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.519683 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.713662 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.713626 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.909748 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:51.909709 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.111349 5973 generic.go:356] "Generic (PLEG): container finished" podID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerID="d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b" exitCode=0 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.111419 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerDied","Data":"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.111852 5973 scope.go:122] "RemoveContainer" containerID="d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.113248 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" event={"ID":"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f","Type":"ContainerStarted","Data":"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.113282 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" event={"ID":"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f","Type":"ContainerStarted","Data":"a44e6bd3c0f167268c733780058ab309c88ba31ec8c6c97c95f47c636e19fe9b"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.113251 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.113345 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.113382 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.139942 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.708777 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:52.724493 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.121291 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerStarted","Data":"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.148069 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" podStartSLOduration=3.148055033 podStartE2EDuration="3.148055033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:52.155704954 +0000 UTC m=+711.930554802" watchObservedRunningTime="2026-11-03 21:57:53.148055033 +0000 UTC m=+712.922904879" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.417533 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.430035 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.442810 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:53.537237 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.023843 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.024688 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.024732 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.131060 5973 generic.go:356] "Generic (PLEG): container finished" podID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerID="c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb" exitCode=0 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.131137 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerDied","Data":"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.131574 5973 scope.go:122] "RemoveContainer" containerID="c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.132974 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.133050 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.133085 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.133315 5973 generic.go:356] "Generic (PLEG): container finished" podID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerID="0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762" exitCode=0 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.133384 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerDied","Data":"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.133896 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.134305 5973 scope.go:122] "RemoveContainer" containerID="0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.136021 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.136097 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.136143 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.150650 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.542710 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.542776 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.544172 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.544220 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.579502 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:57:54.596095 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.579554 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:57:54.947409 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:54.947371 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:57:55.032486 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.027617 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:55.032486 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.027698 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:55.064276 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.064083 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:55.218561 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.217671 5973 generic.go:356] "Generic (PLEG): container finished" podID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerID="c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83" exitCode=0 Nov 03 21:57:55.218561 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.217803 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerDied","Data":"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83"} Nov 03 21:57:55.218561 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.218270 5973 scope.go:122] "RemoveContainer" containerID="c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83" Nov 03 21:57:55.240766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.240725 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.241049 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.241002 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.241115 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.241076 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.245754 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.245729 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerStarted","Data":"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15"} Nov 03 21:57:55.248063 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.248042 5973 generic.go:356] "Generic (PLEG): container finished" podID="c50978f4-39bd-4045-8c75-2ba431f257de" containerID="e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62" exitCode=0 Nov 03 21:57:55.248188 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.248169 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerDied","Data":"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62"} Nov 03 21:57:55.248531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.248514 5973 scope.go:122] "RemoveContainer" containerID="e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62" Nov 03 21:57:55.254509 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.254478 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.254595 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.254558 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.254634 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.254602 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:55.352207 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.352172 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:34560->10.134.0.49:15021: read: connection reset by peer" start-of-body= Nov 03 21:57:55.352345 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.352228 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:34560->10.134.0.49:15021: read: connection reset by peer" Nov 03 21:57:55.488504 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.485230 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerStarted","Data":"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379"} Nov 03 21:57:55.746579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.746507 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:57:55.747544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.747524 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:57:55.747655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.747558 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:57:55.926051 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.925926 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:55.963484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.959638 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:57:55.986484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.967565 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:57:55.986484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:55.967623 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:57:56.067538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.063819 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:56.067538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.063879 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:56.316485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.303093 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:56.478876 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.478793 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:56.606783 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.602818 5973 generic.go:356] "Generic (PLEG): container finished" podID="324bdd22-26a5-4724-9d87-4c369423398d" containerID="5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13" exitCode=0 Nov 03 21:57:56.606783 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.603288 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerDied","Data":"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13"} Nov 03 21:57:56.606783 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.605516 5973 scope.go:122] "RemoveContainer" containerID="5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13" Nov 03 21:57:56.635623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.628212 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:56.635623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.628385 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:56.635623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.628516 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:57:56.724421 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.724382 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerStarted","Data":"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77"} Nov 03 21:57:56.730024 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.729934 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerStarted","Data":"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da"} Nov 03 21:57:56.769577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.769543 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:57:56.770222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.770189 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:57:56.941484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.931433 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:57:56.950623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.950583 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:57:56.950908 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.950881 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:57:56.957222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.955855 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:57:56.957222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:56.955900 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:57:57.037534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.037414 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:57.037534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.037513 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:57.143292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.143254 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 21:57:57.231808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.231768 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:57.583551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.571770 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:57:57.583551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.571896 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:57:57.583551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.571952 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:57:57.636508 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.636411 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:57.852555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.838955 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:57:57.852555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.839016 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:57:57.852555 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.841410 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 21:57:57.903742 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.880167 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 21:57:57.903742 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.880283 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:57.961482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:57.960506 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:57:58.010483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.007313 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bw6\" (UniqueName: \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.232299 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.031104 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.031513 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-53a2cb\"/\"kube-root-ca.crt\"" Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.032565 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-53a2cb\"/\"default-dockercfg-tj995\"" Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.032619 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-53a2cb\"/\"openshift-service-ca.crt\"" Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.032818 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-53a2cb\"/\"simulator-w2-53a2cb-kserve-self-signed-certs\"" Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.149569 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 21:57:58.233127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.191727 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:58.239507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.035594 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.239507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.036067 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.239507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.235157 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerStarted","Data":"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0"} Nov 03 21:57:58.257870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.257835 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:57:58.258194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.232248 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.258671 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.258653 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.286379 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.286349 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.287273 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.258828 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:58.287442 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.258811 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:57:58.287753 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.287731 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.287968 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.287951 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402284 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402338 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402387 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402430 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402514 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bw6\" (UniqueName: \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.402642 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.403126 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.405547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.403378 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.410511 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.407181 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.492574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.492508 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.496846 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.496805 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.572607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.572566 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.572921 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.572870 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:57:58.710585 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.699211 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:58.740495 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.729768 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bw6\" (UniqueName: \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\") pod \"simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.767635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.767576 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.767812 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.767674 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:57:58.797333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.797293 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:57:58.882485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.866563 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.907485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.885553 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:57:58.962012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.961915 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:57:58.965561 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:58.965522 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:57:59.005615 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.005569 5973 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:57:59.029961 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.029923 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.032035 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.031998 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:57:59.120487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.119446 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 21:57:59.177834 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.177799 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:57:59.178157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.178137 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.181563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.181524 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:57:59.284675 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.284562 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:57:59.289485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.285529 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:57:59.566602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.556144 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:59.612173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.611647 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.612173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.611730 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:57:59.933516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.881373 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.933516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.881434 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:57:59.933516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.897599 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.933516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.897655 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:57:59.943764 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.937554 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 21:57:59.979001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.978522 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:57:59.979001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.978582 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:57:59.982668 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:57:59.979799 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:00.146718 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.145578 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 21:58:00.171524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.147057 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:58:00.205198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.180348 5973 patch_prober.go:29] interesting pod/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.205198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.180403 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 21:58:00.205198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.202792 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.205198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.202849 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:00.389017 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.388935 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:00.585483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.565668 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:00.695516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.684439 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.695516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.684556 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:58:00.742488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.724400 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:58:00.758676 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.758505 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.758676 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.758559 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:58:00.891315 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.891277 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" event={"ID":"ab0ed1f9-7468-479c-87ef-b60024d86db4","Type":"ContainerStarted","Data":"3bade80c13a23807ed9c71f7592c32d7ab69b2df4ae77b79eef921ab4de46e57"} Nov 03 21:58:00.893524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.893495 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.893720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.893697 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:58:00.959037 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.958946 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:58:00.959635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:00.959604 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:58:01.080528 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.080444 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:58:01.166494 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.165561 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:01.186529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.168005 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:01.186529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.168070 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:01.187542 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.187501 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:58:01.260709 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.260615 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:01.591544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.589200 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:01.591544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.590878 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:58:01.591544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.590925 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:58:01.735487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.731262 5973 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 03 21:58:01.757485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.748143 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:58:01.757485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.748249 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:58:01.875571 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.871116 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:58:01.875571 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.871195 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:58:01.977483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.974277 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:58:01.988985 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.988399 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:01.989177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.989048 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:58:02.021485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:01.989958 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 21:58:02.037828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.037773 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 21:58:02.041483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.023585 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8af3f3\"/\"simulator-w6-8af3f3-kserve-self-signed-certs\"" Nov 03 21:58:02.041701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.023481 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8af3f3\"/\"default-dockercfg-x4rwd\"" Nov 03 21:58:02.045669 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.023665 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8af3f3\"/\"kube-root-ca.crt\"" Nov 03 21:58:02.045884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.023732 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8af3f3\"/\"openshift-service-ca.crt\"" Nov 03 21:58:02.095734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.095688 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.096076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.096057 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.096256 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.096229 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.096389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.096376 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.096553 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.096533 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml85c\" (UniqueName: \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.096697 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.096681 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.114894 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.114805 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:02.173490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.169515 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:02.187839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.187690 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:02.187839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.187748 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:02.215505 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.212014 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.219940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.219911 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.263683 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.263766 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.263819 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml85c\" (UniqueName: \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.263885 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.264100 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.276484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.269859 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.322583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.322511 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.340539 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.336448 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:02.340539 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.337292 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.365615 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.345771 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.365615 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.365219 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml85c\" (UniqueName: \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\") pod \"simulator-w6-8af3f3-kserve-568b69889b-m4twr\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.367093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.366985 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:02.451165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.450605 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:58:02.593484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.567815 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:58:02.593484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.567875 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:58:02.728538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.717011 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:02.781166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.780639 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 21:58:02.781166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.780700 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 21:58:02.854928 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.854888 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:58:02.855221 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:02.855190 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:58:03.237533 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.176601 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:03.237533 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.176659 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:03.237533 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.177214 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 21:58:03.409485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.395834 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" event={"ID":"ab0ed1f9-7468-479c-87ef-b60024d86db4","Type":"ContainerStarted","Data":"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9"} Nov 03 21:58:03.580574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.566776 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:58:03.580574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.566835 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:58:03.597287 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.594857 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:03.622552 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.620726 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" podStartSLOduration=6.6207044 podStartE2EDuration="6.6207044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:03.522554905 +0000 UTC m=+723.297404753" watchObservedRunningTime="2026-11-03 21:58:03.6207044 +0000 UTC m=+723.395554248" Nov 03 21:58:03.795628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.765753 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:58:03.795628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.767042 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:58:03.795628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.775044 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:58:03.795628 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.783963 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 21:58:03.800602 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.800577 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:58:03.857485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.851810 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:58:03.857485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:03.851865 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:58:04.034623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.024611 5973 patch_prober.go:29] interesting pod/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:58:04.034623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.024716 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:04.166481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.161834 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:04.166481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.161913 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:04.226716 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.221832 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" podUID="e47462ee-ac43-4f06-aeda-0dd21997a7be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:04.251854 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.251824 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:58:04.278829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.274033 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:58:04.305701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.299736 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 21:58:04.408491 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.402105 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:04.458544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.444675 5973 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:58:04.519485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.509667 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" event={"ID":"ace72d8d-5412-43b6-a17c-f3158ef31910","Type":"ContainerStarted","Data":"1d1ae42007a081943294397cb947342abab0b457da193929e6fa3e30a8cb41b9"} Nov 03 21:58:04.576531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.569962 5973 patch_prober.go:29] interesting pod/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 21:58:04.576531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.570050 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 21:58:04.630761 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.626939 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw" Nov 03 21:58:04.850253 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.850214 5973 patch_prober.go:29] interesting pod/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 21:58:04.850411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:04.850283 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 21:58:05.154754 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.154700 5973 patch_prober.go:29] interesting pod/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 21:58:05.154955 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.154766 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 21:58:05.253607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.251897 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:05.452114 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.452015 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:58:05.526484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.525504 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" event={"ID":"ace72d8d-5412-43b6-a17c-f3158ef31910","Type":"ContainerStarted","Data":"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7"} Nov 03 21:58:05.552997 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.552932 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" podStartSLOduration=4.552879063 podStartE2EDuration="4.552879063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:05.546295129 +0000 UTC m=+725.321144978" watchObservedRunningTime="2026-11-03 21:58:05.552879063 +0000 UTC m=+725.327728911" Nov 03 21:58:05.603062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.589826 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:58:05.603062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.591507 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:58:05.603062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:05.592513 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:58:06.152806 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.152767 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:06.159897 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.159872 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:58:06.160204 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.160172 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:58:06.161093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.161077 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 21:58:06.377613 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.377578 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:06.568480 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.568376 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:58:06.569292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.569220 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:58:06.570297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.570278 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 21:58:06.942478 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:06.942412 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:07.761889 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:07.761857 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:07.949061 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:07.949030 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:58:07.964722 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:07.964689 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:58:08.594307 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:08.593912 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:08.798974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:08.798940 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:58:08.811797 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:08.811771 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:58:09.368751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:09.368713 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:09.567852 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:09.567821 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 21:58:10.132440 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:10.132394 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:10.736990 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:10.736957 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:58:10.750188 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:10.750161 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:58:10.801685 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:10.801654 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:11.706569 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:11.706507 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:12.370107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:12.370075 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:12.385640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:12.385607 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:12.537745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:12.537710 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:12.581469 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:12.581433 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:58:13.232652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:13.232618 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:13.850147 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:13.850111 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:14.138502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:14.138402 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:14.399922 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:58:14.399892 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:58:14.835103 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:14.835007 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:15.539104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:15.539065 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:16.427375 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:16.427338 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:17.151899 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:17.151864 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:17.849974 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:17.849939 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:17.974253 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:17.974220 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:58:18.932158 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:18.932110 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:19.650341 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:19.650296 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:20.330350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:20.330299 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:20.750225 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:20.750190 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:21.209276 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:21.209244 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:22.007697 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:22.007664 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:22.658208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:22.658176 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:23.221550 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.221521 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 21:58:23.225003 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.224978 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.233415 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.233371 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc16de\"/\"default-dockercfg-p6w68\"" Nov 03 21:58:23.236434 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.236410 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 21:58:23.236588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.236552 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc16de\"/\"openshift-service-ca.crt\"" Nov 03 21:58:23.236656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.236613 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc16de\"/\"premium-w4-fc16de-kserve-self-signed-certs\"" Nov 03 21:58:23.236656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.236640 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc16de\"/\"kube-root-ca.crt\"" Nov 03 21:58:23.309238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309210 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.309358 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309272 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsnh\" (UniqueName: \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.309399 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309371 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.309438 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309407 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.309568 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309555 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.309607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.309587 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.327990 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.327956 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:23.410807 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410785 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.410914 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410818 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.410914 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410849 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.410914 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410901 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsnh\" (UniqueName: \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.411048 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410919 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.411048 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.410949 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.411201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.411179 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.411254 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.411236 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.411314 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.411299 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.413336 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.413310 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.413449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.413433 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.422842 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.422818 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsnh\" (UniqueName: \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\") pod \"premium-w4-fc16de-kserve-5479d48f79-4v9wc\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.486847 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.486745 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:23.538153 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.538119 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:23.817354 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:23.817306 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 21:58:23.818377 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:58:23.818349 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod862511e9_381b_4710_8587_f0a9abbe9caf.slice/crio-04b588029b33aff2223822d7b6835cc1ccf5ad7391a98e8065e54b1ed520ae35 WatchSource:0}: Error finding container 04b588029b33aff2223822d7b6835cc1ccf5ad7391a98e8065e54b1ed520ae35: Status 404 returned error can't find the container with id 04b588029b33aff2223822d7b6835cc1ccf5ad7391a98e8065e54b1ed520ae35 Nov 03 21:58:24.131976 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:24.131889 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:24.642621 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:24.642569 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" event={"ID":"862511e9-381b-4710-8587-f0a9abbe9caf","Type":"ContainerStarted","Data":"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63"} Nov 03 21:58:24.642998 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:24.642634 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" event={"ID":"862511e9-381b-4710-8587-f0a9abbe9caf","Type":"ContainerStarted","Data":"04b588029b33aff2223822d7b6835cc1ccf5ad7391a98e8065e54b1ed520ae35"} Nov 03 21:58:24.663723 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:24.663678 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" podStartSLOduration=1.663662878 podStartE2EDuration="1.663662878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:24.660862034 +0000 UTC m=+744.435711880" watchObservedRunningTime="2026-11-03 21:58:24.663662878 +0000 UTC m=+744.438512770" Nov 03 21:58:24.947685 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:24.947588 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:25.538849 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:25.538820 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:25.688892 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:25.688854 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:26.274026 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:26.273994 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:26.442653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:26.442616 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:27.437041 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:27.437005 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:27.679906 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:27.679873 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:28.050734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:28.050696 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:28.238701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:28.238671 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:58:28.945916 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:28.945876 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:29.726576 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:29.726543 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:29.913642 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:29.913604 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:30.528537 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:30.528390 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:31.069343 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.069310 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:58:31.074592 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.074570 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.084272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.084248 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8af3f3\"/\"premium-w6-8af3f3-kserve-self-signed-certs\"" Nov 03 21:58:31.084870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.084829 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:58:31.169861 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.169826 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:31.199136 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199102 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.199293 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199149 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsvn\" (UniqueName: \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.199293 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199286 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.199420 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199330 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.199420 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199353 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.199600 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.199582 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300482 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300555 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300578 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300718 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300919 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300757 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.300919 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.300796 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsvn\" (UniqueName: \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.301069 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.301043 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.301114 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.301083 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.301308 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.301289 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.302980 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.302957 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.303358 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.303340 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.313355 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.313328 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsvn\" (UniqueName: \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\") pod \"premium-w6-8af3f3-kserve-85f9df9b89-6xskj\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.390776 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.390696 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:31.444852 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.444818 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:31.678633 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.678611 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:58:31.680628 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:58:31.680592 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c549dd8_99c7_4adf_89eb_8d38dc032d8f.slice/crio-74488b5494237a45a7a7e8d618eb3ec38579afbe55a1efff0b72529986f4b13d WatchSource:0}: Error finding container 74488b5494237a45a7a7e8d618eb3ec38579afbe55a1efff0b72529986f4b13d: Status 404 returned error can't find the container with id 74488b5494237a45a7a7e8d618eb3ec38579afbe55a1efff0b72529986f4b13d Nov 03 21:58:31.689345 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:31.689303 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" event={"ID":"1c549dd8-99c7-4adf-89eb-8d38dc032d8f","Type":"ContainerStarted","Data":"74488b5494237a45a7a7e8d618eb3ec38579afbe55a1efff0b72529986f4b13d"} Nov 03 21:58:32.153788 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:32.149873 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:32.704042 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:32.704000 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" event={"ID":"1c549dd8-99c7-4adf-89eb-8d38dc032d8f","Type":"ContainerStarted","Data":"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351"} Nov 03 21:58:32.724740 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:32.724691 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" podStartSLOduration=1.724673803 podStartE2EDuration="1.724673803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:32.723200644 +0000 UTC m=+752.498050504" watchObservedRunningTime="2026-11-03 21:58:32.724673803 +0000 UTC m=+752.499523655" Nov 03 21:58:32.905541 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:32.905368 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:33.391586 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:33.391555 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:33.539074 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:33.538983 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:33.551543 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:33.551519 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:33.580332 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:33.580299 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:33.721760 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:33.721732 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 21:58:34.339701 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:34.339670 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:35.142051 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:35.142004 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:35.790857 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:35.790824 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:36.535429 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:36.535396 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:37.028623 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.028539 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:58:37.032869 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.032847 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.039283 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.039255 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c0caef-openshift-default-dockercfg-4vqx5\"" Nov 03 21:58:37.045114 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.045085 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:58:37.078933 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.078897 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079137 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.078947 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079137 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079117 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079264 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079160 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079264 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079190 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079264 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079247 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079308 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96cs\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079333 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.079430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.079413 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.209269 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.209222 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.218080 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.210302 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.221551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.221513 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96cs\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.221803 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.221784 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.223288 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.223267 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.223924 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.223902 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.224485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.224447 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.228650 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.228698 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.228725 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.229099 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.224390 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.229373 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.226951 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.234345 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96cs\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.237241 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.243337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.240048 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.245430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.245363 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\") pod \"e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.350292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.350197 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:37.536579 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.536505 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:37.709450 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.709313 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:58:37.758607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.757905 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:58:37.758607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.757998 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:58:37.758607 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.758053 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 21:58:37.776791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:37.776702 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" event={"ID":"b37b4367-bffb-407c-896e-0c2b463fafd4","Type":"ContainerStarted","Data":"450cd65ce81b6bc870de1842b9f02e2ad1d26ce4e0d171338e00039f77a79173"} Nov 03 21:58:38.246327 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:38.245984 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:38.789614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:38.789583 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" event={"ID":"b37b4367-bffb-407c-896e-0c2b463fafd4","Type":"ContainerStarted","Data":"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c"} Nov 03 21:58:38.821547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:38.821499 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podStartSLOduration=1.821485205 podStartE2EDuration="1.821485205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:38.81947644 +0000 UTC m=+758.594326285" watchObservedRunningTime="2026-11-03 21:58:38.821485205 +0000 UTC m=+758.596335049" Nov 03 21:58:39.325630 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:39.325594 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:39.351527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:39.351496 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:39.353120 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:39.353086 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:39.353255 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:39.353149 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:39.645472 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:39.645410 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:40.110129 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:40.110012 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:40.351998 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:40.351969 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:40.352159 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:40.352029 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:40.993218 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:40.993180 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:41.352431 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.352337 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:41.352431 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.352394 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:41.391070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.391040 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:41.404565 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.404541 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:41.820102 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.820072 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:58:41.950879 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:41.950848 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:42.351979 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:42.351947 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:42.352194 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:42.352008 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:42.640549 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:42.637092 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:43.352212 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:43.352177 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:43.352410 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:43.352236 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:43.446467 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:43.446419 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:44.247226 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:44.247191 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:44.352273 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:44.352243 5973 patch_prober.go:29] interesting pod/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 21:58:44.352480 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:44.352306 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 21:58:44.399611 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:58:44.399573 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:58:45.061843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:45.061809 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:45.356221 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:45.356146 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:45.356632 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:45.356432 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:45.364320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:45.364285 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:58:46.206229 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:46.206199 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:47.046105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:47.046072 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:47.736289 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:47.736248 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:48.482193 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:48.482145 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:49.141636 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:49.141600 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:49.347848 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:49.347804 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:58:49.797538 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:49.797505 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:50.834430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:50.834319 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:51.569727 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:51.569684 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:52.445603 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:52.445561 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:53.433738 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:53.433699 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.123653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:54.123611 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.853261 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:54.853220 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:55.653806 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.653772 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j"] Nov 03 21:58:55.657823 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.657802 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.664173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.664149 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 21:58:55.664173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.664173 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:58:55.664359 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.664236 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:58:55.664359 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.664149 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:58:55.668209 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.668187 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j"] Nov 03 21:58:55.726424 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.726386 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.726652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.726516 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.726652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.726535 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2mt\" (UniqueName: \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.776814 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.776782 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:55.830202 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.830165 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.830451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.830427 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2mt\" (UniqueName: \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.830801 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.830778 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.832099 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.832075 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.837395 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.837365 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.844397 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.844378 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2mt\" (UniqueName: \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\") pod \"maas-api-revoke-sub-d7c5cf1b-jfl6j\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:55.973201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:55.973123 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:58:56.263561 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:56.263537 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j"] Nov 03 21:58:56.265175 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:58:56.265137 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6341c596_5a6e_48c2_ba39_fd728b9e7b3e.slice/crio-f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a WatchSource:0}: Error finding container f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a: Status 404 returned error can't find the container with id f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a Nov 03 21:58:56.352713 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:56.352676 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:56.506146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:56.506111 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:57.001177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:57.001070 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" event={"ID":"6341c596-5a6e-48c2-ba39-fd728b9e7b3e","Type":"ContainerStarted","Data":"f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a"} Nov 03 21:58:57.338255 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:57.338140 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:58.062072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.062034 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:58.857617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.857564 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:58.866546 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.866449 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns"] Nov 03 21:58:58.883588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.883557 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns"] Nov 03 21:58:58.883762 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.883688 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:58.985781 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.985748 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:58.985943 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.985881 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:58.986054 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:58.986037 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhvv\" (UniqueName: \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.089105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.089075 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.089540 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.089195 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.089540 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.089320 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhvv\" (UniqueName: \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.090295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.090273 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.092243 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.092213 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.105603 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.105579 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhvv\" (UniqueName: \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\") pod \"maas-api-revoke-sub-51dd77e1-qkxns\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.206400 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.206366 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:58:59.300399 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.300370 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct"] Nov 03 21:58:59.305678 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.305654 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.314001 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.313974 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct"] Nov 03 21:58:59.393808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.393780 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.393926 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.393901 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.393995 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.393943 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrsf\" (UniqueName: \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.492597 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.492552 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns"] Nov 03 21:58:59.493374 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:58:59.493351 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4ee6ae_3404_48f5_9f8a_04d3e33286ae.slice/crio-f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7 WatchSource:0}: Error finding container f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7: Status 404 returned error can't find the container with id f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7 Nov 03 21:58:59.494893 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.494871 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.495010 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.494966 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.495010 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.494996 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrsf\" (UniqueName: \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.495676 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.495657 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.497453 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.497434 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.506751 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.506729 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrsf\" (UniqueName: \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\") pod \"maas-api-revoke-sub-ab2f659d-vlnct\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.600387 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.600349 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:59.623417 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.623381 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:58:59.905521 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:58:59.905497 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct"] Nov 03 21:58:59.907108 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:58:59.907083 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128f001f_d922_4965_a4af_d30c8390962a.slice/crio-493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf WatchSource:0}: Error finding container 493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf: Status 404 returned error can't find the container with id 493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf Nov 03 21:59:00.022368 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.022279 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" event={"ID":"128f001f-d922-4965-a4af-d30c8390962a","Type":"ContainerStarted","Data":"493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf"} Nov 03 21:59:00.023973 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.023946 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" event={"ID":"6341c596-5a6e-48c2-ba39-fd728b9e7b3e","Type":"ContainerStarted","Data":"2771845fb1ad6f32ae388c54fab657fec1933ae86207622d028e27ef9cfbb111"} Nov 03 21:59:00.025684 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.025661 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" event={"ID":"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae","Type":"ContainerStarted","Data":"e4766883484409825a57030c722668cb52b035dfe97f2f6ffe09ca1b6eaf5421"} Nov 03 21:59:00.025809 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.025706 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" event={"ID":"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae","Type":"ContainerStarted","Data":"f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7"} Nov 03 21:59:00.044309 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.044257 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" podStartSLOduration=2.270896592 podStartE2EDuration="5.044243317s" totalImagesPullingTime="2.773346725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:00.041695542 +0000 UTC m=+779.816545392" watchObservedRunningTime="2026-11-03 21:59:00.044243317 +0000 UTC m=+779.819093168" Nov 03 21:59:00.061156 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.061104 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" podStartSLOduration=2.061090013 podStartE2EDuration="2.061090013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:00.059186771 +0000 UTC m=+779.834036619" watchObservedRunningTime="2026-11-03 21:59:00.061090013 +0000 UTC m=+779.835939859" Nov 03 21:59:00.292841 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:00.292751 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:01.033433 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.033342 5973 generic.go:356] "Generic (PLEG): container finished" podID="128f001f-d922-4965-a4af-d30c8390962a" containerID="af953e630a774233a4eb307091990f6c32133dd28b633fbca09a6573a4bcad21" exitCode=0 Nov 03 21:59:01.033617 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.033430 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" event={"ID":"128f001f-d922-4965-a4af-d30c8390962a","Type":"ContainerDied","Data":"af953e630a774233a4eb307091990f6c32133dd28b633fbca09a6573a4bcad21"} Nov 03 21:59:01.034967 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.034944 5973 generic.go:356] "Generic (PLEG): container finished" podID="6341c596-5a6e-48c2-ba39-fd728b9e7b3e" containerID="2771845fb1ad6f32ae388c54fab657fec1933ae86207622d028e27ef9cfbb111" exitCode=0 Nov 03 21:59:01.035104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.035047 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" event={"ID":"6341c596-5a6e-48c2-ba39-fd728b9e7b3e","Type":"ContainerDied","Data":"2771845fb1ad6f32ae388c54fab657fec1933ae86207622d028e27ef9cfbb111"} Nov 03 21:59:01.036451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.036428 5973 generic.go:356] "Generic (PLEG): container finished" podID="8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" containerID="e4766883484409825a57030c722668cb52b035dfe97f2f6ffe09ca1b6eaf5421" exitCode=0 Nov 03 21:59:01.036535 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.036492 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" event={"ID":"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae","Type":"ContainerDied","Data":"e4766883484409825a57030c722668cb52b035dfe97f2f6ffe09ca1b6eaf5421"} Nov 03 21:59:01.061238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:01.061211 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:02.008482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.007932 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:02.432501 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.432055 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:59:02.549952 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.549931 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:59:02.554098 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.554067 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:59:02.592635 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.592538 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\") pod \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " Nov 03 21:59:02.592946 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.592906 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca" pod "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" (UID: "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:02.593066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.593008 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\" (UniqueName: \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\") pod \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " Nov 03 21:59:02.593129 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.593083 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\") pod \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\" (UID: \"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae\") " Nov 03 21:59:02.593636 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.593601 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.596277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.596252 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv" pod "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" (UID: "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae"). InnerVolumeSpecName "kube-api-access-rlhvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:02.596702 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.596682 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca" pod "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" (UID: "8c4ee6ae-3404-48f5-9f8a-04d3e33286ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:02.694349 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694321 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\") pod \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " Nov 03 21:59:02.694545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694366 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\") pod \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " Nov 03 21:59:02.694545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694416 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\" (UniqueName: \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\") pod \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\" (UID: \"6341c596-5a6e-48c2-ba39-fd728b9e7b3e\") " Nov 03 21:59:02.694545 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694526 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\" (UniqueName: \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\") pod \"128f001f-d922-4965-a4af-d30c8390962a\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " Nov 03 21:59:02.694672 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694619 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\") pod \"128f001f-d922-4965-a4af-d30c8390962a\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " Nov 03 21:59:02.694720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694681 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\") pod \"128f001f-d922-4965-a4af-d30c8390962a\" (UID: \"128f001f-d922-4965-a4af-d30c8390962a\") " Nov 03 21:59:02.694817 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.694794 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca" pod "6341c596-5a6e-48c2-ba39-fd728b9e7b3e" (UID: "6341c596-5a6e-48c2-ba39-fd728b9e7b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:02.695121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.695045 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca" pod "128f001f-d922-4965-a4af-d30c8390962a" (UID: "128f001f-d922-4965-a4af-d30c8390962a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:02.695121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.695148 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.695121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.695171 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlhvv\" (UniqueName: \"kubernetes.io/projected/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-kube-api-access-rlhvv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.695121 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.695187 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.695541 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.695236 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128f001f-d922-4965-a4af-d30c8390962a-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.696829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.696806 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt" pod "6341c596-5a6e-48c2-ba39-fd728b9e7b3e" (UID: "6341c596-5a6e-48c2-ba39-fd728b9e7b3e"). InnerVolumeSpecName "kube-api-access-cc2mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:02.696930 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.696828 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca" pod "6341c596-5a6e-48c2-ba39-fd728b9e7b3e" (UID: "6341c596-5a6e-48c2-ba39-fd728b9e7b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:02.696930 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.696845 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca" pod "128f001f-d922-4965-a4af-d30c8390962a" (UID: "128f001f-d922-4965-a4af-d30c8390962a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:02.696930 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.696859 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf" pod "128f001f-d922-4965-a4af-d30c8390962a" (UID: "128f001f-d922-4965-a4af-d30c8390962a"). InnerVolumeSpecName "kube-api-access-6lrsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:02.796130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.796103 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2mt\" (UniqueName: \"kubernetes.io/projected/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-kube-api-access-cc2mt\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.796130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.796128 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lrsf\" (UniqueName: \"kubernetes.io/projected/128f001f-d922-4965-a4af-d30c8390962a-kube-api-access-6lrsf\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.796329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.796140 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128f001f-d922-4965-a4af-d30c8390962a-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.796329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.796150 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6341c596-5a6e-48c2-ba39-fd728b9e7b3e-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:02.836339 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.836305 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:02.989078 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:02.989046 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:03.071310 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.071280 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" Nov 03 21:59:03.071793 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.071338 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns" event={"ID":"8c4ee6ae-3404-48f5-9f8a-04d3e33286ae","Type":"ContainerDied","Data":"f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7"} Nov 03 21:59:03.071793 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.071390 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e9bb4e0b6ce6db7ddb9c1565e633b5e70cdfb7a788304b8d4c3298e16246c7" Nov 03 21:59:03.072874 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.072840 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" Nov 03 21:59:03.073010 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.072840 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct" event={"ID":"128f001f-d922-4965-a4af-d30c8390962a","Type":"ContainerDied","Data":"493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf"} Nov 03 21:59:03.073010 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.072982 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493df05706a804a19bf55c46efe799c3b8a862f02695e71ee1c18f2db4b81fbf" Nov 03 21:59:03.074917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.074895 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" event={"ID":"6341c596-5a6e-48c2-ba39-fd728b9e7b3e","Type":"ContainerDied","Data":"f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a"} Nov 03 21:59:03.075073 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.074944 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j" Nov 03 21:59:03.075073 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.074946 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50d3c000322d6a6cb641d5eca6df58bcde0c2143590c5bed2ec99a05fc70f3a" Nov 03 21:59:03.534065 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:03.534032 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:04.352409 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:04.351450 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:05.228491 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:05.226939 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:05.943588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:05.943538 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:06.750290 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:06.750256 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:07.546636 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:07.546498 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:08.234883 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:08.234845 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:09.057445 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.057394 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:09.200925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.200896 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828"] Nov 03 21:59:09.201360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201347 5973 cpu_manager.go:414] "containerMap: removing container" podUID="8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" containerName="revoke-keys" Nov 03 21:59:09.201414 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201362 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" containerName="revoke-keys" Nov 03 21:59:09.201414 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201384 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6341c596-5a6e-48c2-ba39-fd728b9e7b3e" containerName="revoke-keys" Nov 03 21:59:09.201414 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201389 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6341c596-5a6e-48c2-ba39-fd728b9e7b3e" containerName="revoke-keys" Nov 03 21:59:09.201414 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201396 5973 cpu_manager.go:414] "containerMap: removing container" podUID="128f001f-d922-4965-a4af-d30c8390962a" containerName="revoke-keys" Nov 03 21:59:09.201414 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201401 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128f001f-d922-4965-a4af-d30c8390962a" containerName="revoke-keys" Nov 03 21:59:09.201583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201490 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" containerName="revoke-keys" Nov 03 21:59:09.201583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201501 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6341c596-5a6e-48c2-ba39-fd728b9e7b3e" containerName="revoke-keys" Nov 03 21:59:09.201583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.201506 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="128f001f-d922-4965-a4af-d30c8390962a" containerName="revoke-keys" Nov 03 21:59:09.206779 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.206757 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.212297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.212266 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:59:09.212297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.212276 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:59:09.212297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.212282 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:59:09.212491 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.212323 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 21:59:09.217595 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.217327 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828"] Nov 03 21:59:09.389018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.388939 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.389018 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.388979 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl94f\" (UniqueName: \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.389266 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.389144 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.490627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.490586 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.490813 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.490639 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl94f\" (UniqueName: \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.490813 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.490720 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.491252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.491228 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.493216 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.493194 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.503166 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.503145 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl94f\" (UniqueName: \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\") pod \"maas-api-revoke-sub-b5b31400-lf828\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.523540 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.523515 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:09.801286 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.801256 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828"] Nov 03 21:59:09.802522 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:09.802448 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74433e7_c72b_46e2_9b2f_a64eb8c052ed.slice/crio-77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad WatchSource:0}: Error finding container 77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad: Status 404 returned error can't find the container with id 77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad Nov 03 21:59:09.930148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:09.930120 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:10.123839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:10.123803 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" event={"ID":"f74433e7-c72b-46e2-9b2f-a64eb8c052ed","Type":"ContainerStarted","Data":"f892b79c6c45707d78b3c94a0c54e902171ff5911c18605d3b7dd7e575d818bc"} Nov 03 21:59:10.124280 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:10.123869 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" event={"ID":"f74433e7-c72b-46e2-9b2f-a64eb8c052ed","Type":"ContainerStarted","Data":"77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad"} Nov 03 21:59:10.142658 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:10.142612 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" podStartSLOduration=1.142599273 podStartE2EDuration="1.142599273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:10.141512749 +0000 UTC m=+789.916362597" watchObservedRunningTime="2026-11-03 21:59:10.142599273 +0000 UTC m=+789.917449120" Nov 03 21:59:10.895108 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:10.895068 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:11.131208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.131125 5973 generic.go:356] "Generic (PLEG): container finished" podID="f74433e7-c72b-46e2-9b2f-a64eb8c052ed" containerID="f892b79c6c45707d78b3c94a0c54e902171ff5911c18605d3b7dd7e575d818bc" exitCode=0 Nov 03 21:59:11.131208 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.131180 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" event={"ID":"f74433e7-c72b-46e2-9b2f-a64eb8c052ed","Type":"ContainerDied","Data":"f892b79c6c45707d78b3c94a0c54e902171ff5911c18605d3b7dd7e575d818bc"} Nov 03 21:59:11.219468 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.219424 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6"] Nov 03 21:59:11.223152 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.223133 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.239808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.239693 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6"] Nov 03 21:59:11.317074 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.317031 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.317252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.317088 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.317252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.317212 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7vw\" (UniqueName: \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.418429 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.418401 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7vw\" (UniqueName: \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.418655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.418515 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.418655 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.418548 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.419162 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.419140 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.421127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.421105 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.430727 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.430696 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7vw\" (UniqueName: \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\") pod \"maas-api-revoke-sub-872718cf-4v5c6\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.540238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.540202 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:11.816619 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.816531 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:11.824500 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:11.824450 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6"] Nov 03 21:59:11.829317 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:11.829286 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5007aadd_1499_42d9_9d85_0f7f4c42ce32.slice/crio-3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637 WatchSource:0}: Error finding container 3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637: Status 404 returned error can't find the container with id 3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637 Nov 03 21:59:12.138866 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.138826 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" event={"ID":"5007aadd-1499-42d9-9d85-0f7f4c42ce32","Type":"ContainerStarted","Data":"3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637"} Nov 03 21:59:12.300833 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.300808 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:12.329180 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.329108 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\") pod \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " Nov 03 21:59:12.331541 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.331507 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca" pod "f74433e7-c72b-46e2-9b2f-a64eb8c052ed" (UID: "f74433e7-c72b-46e2-9b2f-a64eb8c052ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:12.430451 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.430408 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\" (UniqueName: \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\") pod \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " Nov 03 21:59:12.430620 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.430503 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\") pod \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\" (UID: \"f74433e7-c72b-46e2-9b2f-a64eb8c052ed\") " Nov 03 21:59:12.430760 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.430741 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:12.430884 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.430865 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca" pod "f74433e7-c72b-46e2-9b2f-a64eb8c052ed" (UID: "f74433e7-c72b-46e2-9b2f-a64eb8c052ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:12.432496 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.432474 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f" pod "f74433e7-c72b-46e2-9b2f-a64eb8c052ed" (UID: "f74433e7-c72b-46e2-9b2f-a64eb8c052ed"). InnerVolumeSpecName "kube-api-access-kl94f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:12.531938 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.531904 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl94f\" (UniqueName: \"kubernetes.io/projected/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-kube-api-access-kl94f\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:12.531938 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.531938 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74433e7-c72b-46e2-9b2f-a64eb8c052ed-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:12.645860 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:12.645829 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:13.148727 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.148695 5973 generic.go:356] "Generic (PLEG): container finished" podID="5007aadd-1499-42d9-9d85-0f7f4c42ce32" containerID="0ccf3083fa57cceac95a34938a48cc1d2ee18fb384566e7ca4ef175413ff7108" exitCode=0 Nov 03 21:59:13.149193 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.148805 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" event={"ID":"5007aadd-1499-42d9-9d85-0f7f4c42ce32","Type":"ContainerDied","Data":"0ccf3083fa57cceac95a34938a48cc1d2ee18fb384566e7ca4ef175413ff7108"} Nov 03 21:59:13.149193 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.148963 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:59:13.150647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.150626 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" Nov 03 21:59:13.150791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.150669 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828" event={"ID":"f74433e7-c72b-46e2-9b2f-a64eb8c052ed","Type":"ContainerDied","Data":"77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad"} Nov 03 21:59:13.150791 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.150716 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b111ce5825796259a3b77299ce8ce0d046f963d7e91bbcd481b3a2eca984ad" Nov 03 21:59:13.607485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:13.607373 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:14.652539 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:14.632729 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:59:14.691497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.690159 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws"] Nov 03 21:59:14.691497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.690899 5973 cpu_manager.go:414] "containerMap: removing container" podUID="f74433e7-c72b-46e2-9b2f-a64eb8c052ed" containerName="revoke-keys" Nov 03 21:59:14.691497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.690918 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74433e7-c72b-46e2-9b2f-a64eb8c052ed" containerName="revoke-keys" Nov 03 21:59:14.691497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.691071 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74433e7-c72b-46e2-9b2f-a64eb8c052ed" containerName="revoke-keys" Nov 03 21:59:14.699827 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.699354 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:14.736560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.708559 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws"] Nov 03 21:59:14.736560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.708678 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.753549 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.737022 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.754657 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\") pod \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.754742 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\") pod \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.754829 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\" (UniqueName: \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\") pod \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\" (UID: \"5007aadd-1499-42d9-9d85-0f7f4c42ce32\") " Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.755077 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67hj\" (UniqueName: \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.755180 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.755212 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.779524 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.766767 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca" pod "5007aadd-1499-42d9-9d85-0f7f4c42ce32" (UID: "5007aadd-1499-42d9-9d85-0f7f4c42ce32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.861036 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw" pod "5007aadd-1499-42d9-9d85-0f7f4c42ce32" (UID: "5007aadd-1499-42d9-9d85-0f7f4c42ce32"). InnerVolumeSpecName "kube-api-access-gz7vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.861119 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca" pod "5007aadd-1499-42d9-9d85-0f7f4c42ce32" (UID: "5007aadd-1499-42d9-9d85-0f7f4c42ce32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.862881 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.862940 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.863063 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67hj\" (UniqueName: \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.863196 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.863216 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5007aadd-1499-42d9-9d85-0f7f4c42ce32-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.863233 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7vw\" (UniqueName: \"kubernetes.io/projected/5007aadd-1499-42d9-9d85-0f7f4c42ce32-kube-api-access-gz7vw\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:14.865483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.864889 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:14.891511 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.866369 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:14.983811 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:14.983752 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67hj\" (UniqueName: \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:15.006536 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.006450 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:15.013610 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.011853 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:15.252492 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.249312 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" event={"ID":"5007aadd-1499-42d9-9d85-0f7f4c42ce32","Type":"ContainerDied","Data":"3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637"} Nov 03 21:59:15.252492 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.249375 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3472515ec6e6088fb2db3599273bfd5ee788f1a69ac27715851e05815b5e4637" Nov 03 21:59:15.252492 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.249449 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6" Nov 03 21:59:15.352149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.352118 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws"] Nov 03 21:59:15.354108 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:15.354080 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e0a2e_a84a_4a7c_b22d_df3cd894e6e7.slice/crio-85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1 WatchSource:0}: Error finding container 85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1: Status 404 returned error can't find the container with id 85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1 Nov 03 21:59:15.520484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.519606 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz"] Nov 03 21:59:15.520484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.520375 5973 cpu_manager.go:414] "containerMap: removing container" podUID="5007aadd-1499-42d9-9d85-0f7f4c42ce32" containerName="revoke-keys" Nov 03 21:59:15.520484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.520393 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5007aadd-1499-42d9-9d85-0f7f4c42ce32" containerName="revoke-keys" Nov 03 21:59:15.520788 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.520555 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="5007aadd-1499-42d9-9d85-0f7f4c42ce32" containerName="revoke-keys" Nov 03 21:59:15.529583 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.528166 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.534415 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.534283 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz"] Nov 03 21:59:15.590694 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.590658 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.591295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.591271 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.601529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.591589 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.733482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.723817 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.733482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.723893 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.733482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.724032 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.733482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.725983 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.733482 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.731890 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:15.747544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:15.746816 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\") pod \"maas-api-revoke-sub-12d76ccb-xm6dz\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:16.028653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.028618 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:16.372534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.355252 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" event={"ID":"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7","Type":"ContainerStarted","Data":"7977847a99b8a6f8258009b2133913cdad8027ed948ce3094f7ec23f946a6c73"} Nov 03 21:59:16.372534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.355311 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" event={"ID":"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7","Type":"ContainerStarted","Data":"85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1"} Nov 03 21:59:16.623531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.604551 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:59:16.623531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.604616 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:59:16.734481 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.733712 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:16.852743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.848715 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" podStartSLOduration=2.848693062 podStartE2EDuration="2.848693062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:16.497736322 +0000 UTC m=+796.272586169" watchObservedRunningTime="2026-11-03 21:59:16.848693062 +0000 UTC m=+796.623542910" Nov 03 21:59:16.882107 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:16.857912 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz"] Nov 03 21:59:17.353160 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.353120 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8"] Nov 03 21:59:17.364696 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.364652 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.365262 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.365217 5973 generic.go:356] "Generic (PLEG): container finished" podID="d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" containerID="7977847a99b8a6f8258009b2133913cdad8027ed948ce3094f7ec23f946a6c73" exitCode=0 Nov 03 21:59:17.365367 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.365299 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" event={"ID":"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7","Type":"ContainerDied","Data":"7977847a99b8a6f8258009b2133913cdad8027ed948ce3094f7ec23f946a6c73"} Nov 03 21:59:17.372767 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.371423 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8"] Nov 03 21:59:17.377843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.377540 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" event={"ID":"9de99fca-5d51-4b4b-beb5-be19b67ece7f","Type":"ContainerStarted","Data":"1c5e5ef45b79ce392901eb88d2e22ffb982c4d5867ac702f12488ec391e29395"} Nov 03 21:59:17.488358 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.488318 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.488577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.488382 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjjw\" (UniqueName: \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.488577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.488488 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.589322 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.589237 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjjw\" (UniqueName: \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.589322 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.589274 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.589589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.589407 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.589999 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.589971 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.591810 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.591789 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.601684 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.601661 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjjw\" (UniqueName: \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\") pod \"maas-api-revoke-sub-8fd73d6f-hq4h8\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.690335 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.690308 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:17.969923 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:17.969889 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8"] Nov 03 21:59:17.970701 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:17.970677 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98bc69c_1cdc_493b_aab7_df5ecd79fc37.slice/crio-eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7 WatchSource:0}: Error finding container eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7: Status 404 returned error can't find the container with id eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7 Nov 03 21:59:18.302638 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.302614 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:18.385836 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.385802 5973 generic.go:356] "Generic (PLEG): container finished" podID="9de99fca-5d51-4b4b-beb5-be19b67ece7f" containerID="dd2abc69d5f0c319f0b9d431b7bf01fac3834f3d8a9d5bf018c5803202125634" exitCode=0 Nov 03 21:59:18.385988 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.385888 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" event={"ID":"9de99fca-5d51-4b4b-beb5-be19b67ece7f","Type":"ContainerDied","Data":"dd2abc69d5f0c319f0b9d431b7bf01fac3834f3d8a9d5bf018c5803202125634"} Nov 03 21:59:18.387860 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.387825 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" event={"ID":"d98bc69c-1cdc-493b-aab7-df5ecd79fc37","Type":"ContainerStarted","Data":"053d54a66863c0d6d4f3714f9596d8bd48bcb7114a97f2dff6e1ce4ba2070eee"} Nov 03 21:59:18.387993 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.387882 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" event={"ID":"d98bc69c-1cdc-493b-aab7-df5ecd79fc37","Type":"ContainerStarted","Data":"eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7"} Nov 03 21:59:18.426789 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.426676 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" podStartSLOduration=1.4266592390000001 podStartE2EDuration="1.426659239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:18.425327041 +0000 UTC m=+798.200176891" watchObservedRunningTime="2026-11-03 21:59:18.426659239 +0000 UTC m=+798.201509088" Nov 03 21:59:18.510743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.510719 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:18.598835 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.598809 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\") pod \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " Nov 03 21:59:18.599025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.598902 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\") pod \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " Nov 03 21:59:18.599025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.598972 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\" (UniqueName: \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\") pod \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\" (UID: \"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7\") " Nov 03 21:59:18.599270 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.599249 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca" pod "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" (UID: "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:18.599355 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.599342 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:18.601115 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.601083 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca" pod "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" (UID: "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:18.601204 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.601150 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj" pod "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" (UID: "d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7"). InnerVolumeSpecName "kube-api-access-j67hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:18.700350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.700283 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:18.700350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.700309 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67hj\" (UniqueName: \"kubernetes.io/projected/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7-kube-api-access-j67hj\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:18.858206 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:18.858161 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:19.188991 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.188956 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:19.395559 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.395524 5973 generic.go:356] "Generic (PLEG): container finished" podID="d98bc69c-1cdc-493b-aab7-df5ecd79fc37" containerID="053d54a66863c0d6d4f3714f9596d8bd48bcb7114a97f2dff6e1ce4ba2070eee" exitCode=0 Nov 03 21:59:19.395707 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.395612 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" event={"ID":"d98bc69c-1cdc-493b-aab7-df5ecd79fc37","Type":"ContainerDied","Data":"053d54a66863c0d6d4f3714f9596d8bd48bcb7114a97f2dff6e1ce4ba2070eee"} Nov 03 21:59:19.397033 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.396992 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" Nov 03 21:59:19.397033 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.397026 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws" event={"ID":"d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7","Type":"ContainerDied","Data":"85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1"} Nov 03 21:59:19.397183 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.397060 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85ac759bd9bebf1424efd6caa2a048e62481ba8afbb9570c715aadbd68444ad1" Nov 03 21:59:19.555626 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.555595 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:19.635136 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.635093 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:19.712684 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.712596 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\") pod \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " Nov 03 21:59:19.712684 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.712649 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\") pod \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " Nov 03 21:59:19.712906 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.712743 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\") pod \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\" (UID: \"9de99fca-5d51-4b4b-beb5-be19b67ece7f\") " Nov 03 21:59:19.713120 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.713090 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca" pod "9de99fca-5d51-4b4b-beb5-be19b67ece7f" (UID: "9de99fca-5d51-4b4b-beb5-be19b67ece7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:19.713269 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.713249 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:19.715153 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.715130 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca" pod "9de99fca-5d51-4b4b-beb5-be19b67ece7f" (UID: "9de99fca-5d51-4b4b-beb5-be19b67ece7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:19.715416 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.715395 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s" pod "9de99fca-5d51-4b4b-beb5-be19b67ece7f" (UID: "9de99fca-5d51-4b4b-beb5-be19b67ece7f"). InnerVolumeSpecName "kube-api-access-4799s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:19.814506 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.814443 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/9de99fca-5d51-4b4b-beb5-be19b67ece7f-kube-api-access-4799s\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:19.814506 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:19.814508 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de99fca-5d51-4b4b-beb5-be19b67ece7f-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:20.346531 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.346443 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:20.405625 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.405598 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" Nov 03 21:59:20.405975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.405651 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz" event={"ID":"9de99fca-5d51-4b4b-beb5-be19b67ece7f","Type":"ContainerDied","Data":"1c5e5ef45b79ce392901eb88d2e22ffb982c4d5867ac702f12488ec391e29395"} Nov 03 21:59:20.405975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.405724 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5e5ef45b79ce392901eb88d2e22ffb982c4d5867ac702f12488ec391e29395" Nov 03 21:59:20.551633 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.551610 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:20.625544 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.625447 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\") pod \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " Nov 03 21:59:20.625682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.625586 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\" (UniqueName: \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\") pod \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " Nov 03 21:59:20.625725 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.625687 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\") pod \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\" (UID: \"d98bc69c-1cdc-493b-aab7-df5ecd79fc37\") " Nov 03 21:59:20.626135 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.626031 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca" pod "d98bc69c-1cdc-493b-aab7-df5ecd79fc37" (UID: "d98bc69c-1cdc-493b-aab7-df5ecd79fc37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:20.627624 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.627594 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca" pod "d98bc69c-1cdc-493b-aab7-df5ecd79fc37" (UID: "d98bc69c-1cdc-493b-aab7-df5ecd79fc37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:20.627754 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.627737 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw" pod "d98bc69c-1cdc-493b-aab7-df5ecd79fc37" (UID: "d98bc69c-1cdc-493b-aab7-df5ecd79fc37"). InnerVolumeSpecName "kube-api-access-lwjjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:20.725931 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.725900 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:20.727232 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.727214 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:20.727329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.727235 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:20.727329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.727245 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjjw\" (UniqueName: \"kubernetes.io/projected/d98bc69c-1cdc-493b-aab7-df5ecd79fc37-kube-api-access-lwjjw\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.815614 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82"] Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816619 5973 cpu_manager.go:414] "containerMap: removing container" podUID="d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816647 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816659 5973 cpu_manager.go:414] "containerMap: removing container" podUID="9de99fca-5d51-4b4b-beb5-be19b67ece7f" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816666 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de99fca-5d51-4b4b-beb5-be19b67ece7f" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816728 5973 cpu_manager.go:414] "containerMap: removing container" podUID="d98bc69c-1cdc-493b-aab7-df5ecd79fc37" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816736 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98bc69c-1cdc-493b-aab7-df5ecd79fc37" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.816993 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.817005 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de99fca-5d51-4b4b-beb5-be19b67ece7f" containerName="revoke-keys" Nov 03 21:59:20.820374 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.817027 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98bc69c-1cdc-493b-aab7-df5ecd79fc37" containerName="revoke-keys" Nov 03 21:59:20.821634 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.821609 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:20.828249 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.828225 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82"] Nov 03 21:59:20.929441 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.929409 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:20.929627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.929452 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:20.929627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:20.929596 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfxw\" (UniqueName: \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.030843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.030809 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.031025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.030865 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.031025 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.030906 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfxw\" (UniqueName: \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.031571 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.031542 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.033334 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.033313 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.043330 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.043301 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfxw\" (UniqueName: \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.137669 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.137636 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:21.413589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.413564 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" Nov 03 21:59:21.413977 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.413592 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8" event={"ID":"d98bc69c-1cdc-493b-aab7-df5ecd79fc37","Type":"ContainerDied","Data":"eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7"} Nov 03 21:59:21.413977 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.413640 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb7769c3edfff9477ea39ea7928dc5f6ecc358790d1954f8d9caf9ca3521c4c7" Nov 03 21:59:21.421113 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.421078 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82"] Nov 03 21:59:21.421660 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:21.421634 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8557be75_c443_4a89_bfc7_c00e702eb0bb.slice/crio-bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04 WatchSource:0}: Error finding container bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04: Status 404 returned error can't find the container with id bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04 Nov 03 21:59:21.498126 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.497442 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:21.902178 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.902142 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:21.997941 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:21.997907 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78"] Nov 03 21:59:22.001926 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.001901 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.011671 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.011646 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78"] Nov 03 21:59:22.144320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.144288 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.144499 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.144360 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fm4z\" (UniqueName: \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.144499 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.144443 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.245259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.245182 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.245430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.245272 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fm4z\" (UniqueName: \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.245430 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.245295 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.245917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.245898 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.247680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.247654 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.260140 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.260112 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fm4z\" (UniqueName: \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\") pod \"maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.318089 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.318067 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:22.425304 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.425268 5973 generic.go:356] "Generic (PLEG): container finished" podID="8557be75-c443-4a89-bfc7-c00e702eb0bb" containerID="1dcfae3b8fc5e66643dedf9705fdbc77f69e140bdd8926de52f5c21b857c78d7" exitCode=0 Nov 03 21:59:22.425772 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.425362 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" event={"ID":"8557be75-c443-4a89-bfc7-c00e702eb0bb","Type":"ContainerDied","Data":"1dcfae3b8fc5e66643dedf9705fdbc77f69e140bdd8926de52f5c21b857c78d7"} Nov 03 21:59:22.425772 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.425417 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" event={"ID":"8557be75-c443-4a89-bfc7-c00e702eb0bb","Type":"ContainerStarted","Data":"bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04"} Nov 03 21:59:22.601112 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.601085 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78"] Nov 03 21:59:22.601735 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:22.601708 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5532faed_f1c4_4d1a_ac00_87e8125f7f3d.slice/crio-6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69 WatchSource:0}: Error finding container 6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69: Status 404 returned error can't find the container with id 6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69 Nov 03 21:59:22.694244 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:22.694209 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:23.241890 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.241850 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:23.433642 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.433609 5973 generic.go:356] "Generic (PLEG): container finished" podID="5532faed-f1c4-4d1a-ac00-87e8125f7f3d" containerID="c548a3379038b1dde4d15b7a34c618d3456c29ce5cff537e164747ddc93cdfee" exitCode=0 Nov 03 21:59:23.433971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.433680 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" event={"ID":"5532faed-f1c4-4d1a-ac00-87e8125f7f3d","Type":"ContainerDied","Data":"c548a3379038b1dde4d15b7a34c618d3456c29ce5cff537e164747ddc93cdfee"} Nov 03 21:59:23.433971 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.433742 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" event={"ID":"5532faed-f1c4-4d1a-ac00-87e8125f7f3d","Type":"ContainerStarted","Data":"6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69"} Nov 03 21:59:23.606706 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.606681 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:23.648862 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.648826 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:23.762487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.762443 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:59:23.762763 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.762736 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\") pod \"8557be75-c443-4a89-bfc7-c00e702eb0bb\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " Nov 03 21:59:23.762865 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.762846 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\") pod \"8557be75-c443-4a89-bfc7-c00e702eb0bb\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " Nov 03 21:59:23.762976 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.762959 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\" (UniqueName: \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\") pod \"8557be75-c443-4a89-bfc7-c00e702eb0bb\" (UID: \"8557be75-c443-4a89-bfc7-c00e702eb0bb\") " Nov 03 21:59:23.763223 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.763200 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca" pod "8557be75-c443-4a89-bfc7-c00e702eb0bb" (UID: "8557be75-c443-4a89-bfc7-c00e702eb0bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:23.763408 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.763392 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:23.765155 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.765109 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca" pod "8557be75-c443-4a89-bfc7-c00e702eb0bb" (UID: "8557be75-c443-4a89-bfc7-c00e702eb0bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:23.765381 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.765352 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw" pod "8557be75-c443-4a89-bfc7-c00e702eb0bb" (UID: "8557be75-c443-4a89-bfc7-c00e702eb0bb"). InnerVolumeSpecName "kube-api-access-2wfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:23.864398 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.864324 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfxw\" (UniqueName: \"kubernetes.io/projected/8557be75-c443-4a89-bfc7-c00e702eb0bb-kube-api-access-2wfxw\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:23.864398 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.864355 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8557be75-c443-4a89-bfc7-c00e702eb0bb-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:23.896311 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:23.896267 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:24.440723 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.440690 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" event={"ID":"8557be75-c443-4a89-bfc7-c00e702eb0bb","Type":"ContainerDied","Data":"bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04"} Nov 03 21:59:24.441101 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.440747 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbbf26c559ddf66298d1c28a4aab15d44f61c189c300ae26137b2c4ec5859e04" Nov 03 21:59:24.441101 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.440709 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82" Nov 03 21:59:24.598758 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.598735 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:24.710691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.710659 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:24.774942 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.774914 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\") pod \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " Nov 03 21:59:24.775146 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.774978 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\") pod \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " Nov 03 21:59:24.775211 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.775159 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\" (UniqueName: \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\") pod \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\" (UID: \"5532faed-f1c4-4d1a-ac00-87e8125f7f3d\") " Nov 03 21:59:24.775473 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.775424 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca" pod "5532faed-f1c4-4d1a-ac00-87e8125f7f3d" (UID: "5532faed-f1c4-4d1a-ac00-87e8125f7f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:24.775613 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.775593 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:24.776745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.776699 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:59:24.777278 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.777248 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca" pod "5532faed-f1c4-4d1a-ac00-87e8125f7f3d" (UID: "5532faed-f1c4-4d1a-ac00-87e8125f7f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:24.777554 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.777532 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z" pod "5532faed-f1c4-4d1a-ac00-87e8125f7f3d" (UID: "5532faed-f1c4-4d1a-ac00-87e8125f7f3d"). InnerVolumeSpecName "kube-api-access-8fm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:24.877084 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.877055 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fm4z\" (UniqueName: \"kubernetes.io/projected/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-kube-api-access-8fm4z\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:24.877084 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.877083 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5532faed-f1c4-4d1a-ac00-87e8125f7f3d-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:24.910708 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:24.910673 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:25.074440 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.074293 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:25.450356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.448959 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" event={"ID":"5532faed-f1c4-4d1a-ac00-87e8125f7f3d","Type":"ContainerDied","Data":"6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69"} Nov 03 21:59:25.450356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.449025 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bda54bbaefdaa4b002c296073169744b1a5df932857c814ce81ebae83b5df69" Nov 03 21:59:25.450356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.449099 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78" Nov 03 21:59:25.464574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.464099 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:25.632484 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.632429 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:59:25.637887 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.637304 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:59:25.772734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:25.772641 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:26.349231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:26.349002 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:26.810215 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:26.810181 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:59:26.962997 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:26.962963 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:27.121646 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.121566 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:27.592578 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.592544 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq"] Nov 03 21:59:27.593064 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593051 5973 cpu_manager.go:414] "containerMap: removing container" podUID="5532faed-f1c4-4d1a-ac00-87e8125f7f3d" containerName="revoke-keys" Nov 03 21:59:27.593105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593066 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5532faed-f1c4-4d1a-ac00-87e8125f7f3d" containerName="revoke-keys" Nov 03 21:59:27.593105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593077 5973 cpu_manager.go:414] "containerMap: removing container" podUID="8557be75-c443-4a89-bfc7-c00e702eb0bb" containerName="revoke-keys" Nov 03 21:59:27.593105 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593082 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8557be75-c443-4a89-bfc7-c00e702eb0bb" containerName="revoke-keys" Nov 03 21:59:27.593205 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593188 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="5532faed-f1c4-4d1a-ac00-87e8125f7f3d" containerName="revoke-keys" Nov 03 21:59:27.593205 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.593199 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="8557be75-c443-4a89-bfc7-c00e702eb0bb" containerName="revoke-keys" Nov 03 21:59:27.596720 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.596705 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.602516 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.602493 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:59:27.602645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.602571 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:59:27.602645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.602585 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:59:27.602645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.602499 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 21:59:27.606950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.606927 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq"] Nov 03 21:59:27.614646 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.614623 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wdk\" (UniqueName: \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.614770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.614668 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.614770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.614741 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.715331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.715289 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.715526 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.715361 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.715625 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.715568 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wdk\" (UniqueName: \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.716205 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.716182 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.718087 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.718063 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.727973 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.727950 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wdk\" (UniqueName: \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\") pod \"maas-api-revoke-sub-659e34db-nxxjq\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:27.913193 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:27.913161 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:28.227927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.221310 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:28.238356 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.238320 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:59:28.378438 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.378409 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq"] Nov 03 21:59:28.429881 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.429796 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:28.473472 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.473422 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" event={"ID":"db71e8ed-c217-4c3d-a337-01e5f76a991c","Type":"ContainerStarted","Data":"f48f806ba4f65190f196598454ca541669235dba65bb19a5aab9ac52efe9a11b"} Nov 03 21:59:28.842392 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.842310 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d"] Nov 03 21:59:28.847062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.847043 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.852243 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.852219 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt65j\" (UniqueName: \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.852436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.852409 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.852664 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.852543 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.859984 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.859388 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d"] Nov 03 21:59:28.953855 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.953821 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt65j\" (UniqueName: \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.954450 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.954422 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.954653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.954633 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.955156 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.955132 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.970487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.968561 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:28.973140 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:28.973106 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt65j\" (UniqueName: \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\") pod \"maas-api-revoke-sub-dc3913c2-dn64d\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:29.247282 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:29.246792 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:29.358400 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:29.356773 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:29.680499 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:29.619085 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:29.786766 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:29.786667 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" containerID="cri-o://af74429d511493ed6f7a61d9e9504f7b07e649e96b475f37a7b9a0674eeb9f91" gracePeriod=25 Nov 03 21:59:29.853032 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:29.852098 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" event={"ID":"db71e8ed-c217-4c3d-a337-01e5f76a991c","Type":"ContainerStarted","Data":"3cfd3bd37da7dbedbe79c7d6a7641b11e3409c0f3c5e7bcebf92bfec3d66914c"} Nov 03 21:59:30.048714 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.048640 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d"] Nov 03 21:59:30.058525 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:30.058496 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0e0b23_f5be_4b1f_988b_d6c0045c10b5.slice/crio-84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d WatchSource:0}: Error finding container 84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d: Status 404 returned error can't find the container with id 84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d Nov 03 21:59:30.063076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.063021 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" podStartSLOduration=3.063004504 podStartE2EDuration="3.063004504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:30.059923465 +0000 UTC m=+809.834773312" watchObservedRunningTime="2026-11-03 21:59:30.063004504 +0000 UTC m=+809.837854351" Nov 03 21:59:30.864428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.864333 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" event={"ID":"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5","Type":"ContainerStarted","Data":"1aa9186a70cbdf45b74f84736aac2108437a3f772944c1434f1b5523e0959545"} Nov 03 21:59:30.865013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.864985 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" event={"ID":"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5","Type":"ContainerStarted","Data":"84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d"} Nov 03 21:59:30.868257 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.868235 5973 generic.go:356] "Generic (PLEG): container finished" podID="db71e8ed-c217-4c3d-a337-01e5f76a991c" containerID="3cfd3bd37da7dbedbe79c7d6a7641b11e3409c0f3c5e7bcebf92bfec3d66914c" exitCode=0 Nov 03 21:59:30.868431 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.868419 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" event={"ID":"db71e8ed-c217-4c3d-a337-01e5f76a991c","Type":"ContainerDied","Data":"3cfd3bd37da7dbedbe79c7d6a7641b11e3409c0f3c5e7bcebf92bfec3d66914c"} Nov 03 21:59:30.920587 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.910204 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" podStartSLOduration=2.910183205 podStartE2EDuration="2.910183205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:30.893667652 +0000 UTC m=+810.668517501" watchObservedRunningTime="2026-11-03 21:59:30.910183205 +0000 UTC m=+810.685033056" Nov 03 21:59:30.965335 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:30.965299 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.203772 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:31.203732 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.357127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:31.357091 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.757364 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:31.757319 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.877247 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:31.877211 5973 generic.go:356] "Generic (PLEG): container finished" podID="fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" containerID="1aa9186a70cbdf45b74f84736aac2108437a3f772944c1434f1b5523e0959545" exitCode=0 Nov 03 21:59:31.877434 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:31.877299 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" event={"ID":"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5","Type":"ContainerDied","Data":"1aa9186a70cbdf45b74f84736aac2108437a3f772944c1434f1b5523e0959545"} Nov 03 21:59:32.053608 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.053583 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:32.150785 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.150743 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\" (UniqueName: \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\") pod \"db71e8ed-c217-4c3d-a337-01e5f76a991c\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " Nov 03 21:59:32.150785 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.150790 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\") pod \"db71e8ed-c217-4c3d-a337-01e5f76a991c\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " Nov 03 21:59:32.151022 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.150886 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\") pod \"db71e8ed-c217-4c3d-a337-01e5f76a991c\" (UID: \"db71e8ed-c217-4c3d-a337-01e5f76a991c\") " Nov 03 21:59:32.151202 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.151177 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca" pod "db71e8ed-c217-4c3d-a337-01e5f76a991c" (UID: "db71e8ed-c217-4c3d-a337-01e5f76a991c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:32.151354 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.151333 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:32.153287 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.153264 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca" pod "db71e8ed-c217-4c3d-a337-01e5f76a991c" (UID: "db71e8ed-c217-4c3d-a337-01e5f76a991c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:32.153413 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.153352 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk" pod "db71e8ed-c217-4c3d-a337-01e5f76a991c" (UID: "db71e8ed-c217-4c3d-a337-01e5f76a991c"). InnerVolumeSpecName "kube-api-access-v6wdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:32.252297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.252265 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db71e8ed-c217-4c3d-a337-01e5f76a991c-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:32.252297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.252296 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wdk\" (UniqueName: \"kubernetes.io/projected/db71e8ed-c217-4c3d-a337-01e5f76a991c-kube-api-access-v6wdk\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:32.293203 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.293108 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:32.361070 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.361043 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9"] Nov 03 21:59:32.361598 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.361579 5973 cpu_manager.go:414] "containerMap: removing container" podUID="db71e8ed-c217-4c3d-a337-01e5f76a991c" containerName="revoke-keys" Nov 03 21:59:32.361704 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.361599 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db71e8ed-c217-4c3d-a337-01e5f76a991c" containerName="revoke-keys" Nov 03 21:59:32.361768 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.361754 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="db71e8ed-c217-4c3d-a337-01e5f76a991c" containerName="revoke-keys" Nov 03 21:59:32.365535 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.365512 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.369948 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.369924 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv"] Nov 03 21:59:32.375444 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.375420 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.382125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.380402 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9"] Nov 03 21:59:32.392094 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.392072 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv"] Nov 03 21:59:32.468227 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.467021 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:32.560343 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560262 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.560343 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560313 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8bd\" (UniqueName: \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.560546 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560386 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcsn\" (UniqueName: \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.560546 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560510 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.560613 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560557 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.560613 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.560587 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.662113 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662075 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.662113 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662117 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8bd\" (UniqueName: \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.662490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662139 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcsn\" (UniqueName: \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.662490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662195 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.662490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662227 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.662490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662254 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.662947 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.662920 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.663075 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.663012 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.664656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.664631 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.664920 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.664897 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.674304 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.674281 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcsn\" (UniqueName: \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\") pod \"maas-api-revoke-sub-5a4aeb0e-rdqr9\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.674407 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.674337 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8bd\" (UniqueName: \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\") pod \"maas-api-revoke-sub-d902e139-wftrv\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.697431 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.697406 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:32.703639 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.703606 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:32.850114 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.850036 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:32.885917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.885871 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" event={"ID":"db71e8ed-c217-4c3d-a337-01e5f76a991c","Type":"ContainerDied","Data":"f48f806ba4f65190f196598454ca541669235dba65bb19a5aab9ac52efe9a11b"} Nov 03 21:59:32.885917 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.885902 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq" Nov 03 21:59:32.886112 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.885931 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48f806ba4f65190f196598454ca541669235dba65bb19a5aab9ac52efe9a11b" Nov 03 21:59:32.985022 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:32.984987 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9"] Nov 03 21:59:32.989612 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:32.989587 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3224cbfb_42f0_45bd_a293_becc7d2cb02e.slice/crio-b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99 WatchSource:0}: Error finding container b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99: Status 404 returned error can't find the container with id b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99 Nov 03 21:59:33.004839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.004809 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv"] Nov 03 21:59:33.008711 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:33.008684 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568ba166_ca39_459f_94d8_b0ded051adc9.slice/crio-d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2 WatchSource:0}: Error finding container d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2: Status 404 returned error can't find the container with id d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2 Nov 03 21:59:33.015556 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.015532 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:33.170828 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.170799 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\") pod \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " Nov 03 21:59:33.170991 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.170897 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\") pod \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " Nov 03 21:59:33.171072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.171057 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\" (UniqueName: \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\") pod \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\" (UID: \"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5\") " Nov 03 21:59:33.171273 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.171243 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca" pod "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" (UID: "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:33.171492 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.171472 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:33.173122 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.173093 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca" pod "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" (UID: "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:33.173340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.173324 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j" pod "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" (UID: "fe0e0b23-f5be-4b1f-988b-d6c0045c10b5"). InnerVolumeSpecName "kube-api-access-jt65j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:33.272277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.272247 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt65j\" (UniqueName: \"kubernetes.io/projected/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-kube-api-access-jt65j\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:33.272277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.272275 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:33.521633 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.521542 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d"] Nov 03 21:59:33.522312 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.522285 5973 cpu_manager.go:414] "containerMap: removing container" podUID="fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" containerName="revoke-keys" Nov 03 21:59:33.522312 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.522306 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" containerName="revoke-keys" Nov 03 21:59:33.522517 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.522485 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" containerName="revoke-keys" Nov 03 21:59:33.526233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.526211 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.553503 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.552538 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d"] Nov 03 21:59:33.576290 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.576255 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.576443 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.576314 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69msb\" (UniqueName: \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.576443 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.576363 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.607785 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.607747 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:33.677402 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.677377 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.677548 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.677421 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69msb\" (UniqueName: \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.677548 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.677489 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.678198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.678173 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.680201 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.680180 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.689686 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.689663 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69msb\" (UniqueName: \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\") pod \"maas-api-revoke-sub-82911aa3-z2f4d\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.763049 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.763016 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:33.862896 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.862861 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:33.894252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.894218 5973 generic.go:356] "Generic (PLEG): container finished" podID="3224cbfb-42f0-45bd-a293-becc7d2cb02e" containerID="9aea7563973bf811a72b1f4c1f65cdbe653351c07a288fd31ca9d773aba06cc5" exitCode=0 Nov 03 21:59:33.894390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.894275 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" event={"ID":"3224cbfb-42f0-45bd-a293-becc7d2cb02e","Type":"ContainerDied","Data":"9aea7563973bf811a72b1f4c1f65cdbe653351c07a288fd31ca9d773aba06cc5"} Nov 03 21:59:33.894390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.894331 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" event={"ID":"3224cbfb-42f0-45bd-a293-becc7d2cb02e","Type":"ContainerStarted","Data":"b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99"} Nov 03 21:59:33.895901 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.895876 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" Nov 03 21:59:33.896002 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.895921 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d" event={"ID":"fe0e0b23-f5be-4b1f-988b-d6c0045c10b5","Type":"ContainerDied","Data":"84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d"} Nov 03 21:59:33.896002 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.895962 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84bcc43d18ab8634f6879b2dadd5bb915f4bbc019ef2f661310e5b812520832d" Nov 03 21:59:33.897767 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.897737 5973 generic.go:356] "Generic (PLEG): container finished" podID="568ba166-ca39-459f-94d8-b0ded051adc9" containerID="50942e6458902c752bffbd2bbc4a29b38c69a21d1547f5e4138d80fd7369a9e4" exitCode=0 Nov 03 21:59:33.898157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.897813 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" event={"ID":"568ba166-ca39-459f-94d8-b0ded051adc9","Type":"ContainerDied","Data":"50942e6458902c752bffbd2bbc4a29b38c69a21d1547f5e4138d80fd7369a9e4"} Nov 03 21:59:33.898157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.897853 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" event={"ID":"568ba166-ca39-459f-94d8-b0ded051adc9","Type":"ContainerStarted","Data":"d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2"} Nov 03 21:59:33.921645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:33.921620 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:34.142940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.142910 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d"] Nov 03 21:59:34.144035 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:34.144006 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5185f19_313f_414f_b642_c9525ae8adfa.slice/crio-4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9 WatchSource:0}: Error finding container 4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9: Status 404 returned error can't find the container with id 4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9 Nov 03 21:59:34.629432 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.629396 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:59:34.741679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.741564 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:34.906104 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.906071 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" event={"ID":"e5185f19-313f-414f-b642-c9525ae8adfa","Type":"ContainerStarted","Data":"92add4f36332dfc62b666f82eda49d4051d4be64d26bc8d7cd6b51d9ddd03af6"} Nov 03 21:59:34.906259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.906151 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" event={"ID":"e5185f19-313f-414f-b642-c9525ae8adfa","Type":"ContainerStarted","Data":"4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9"} Nov 03 21:59:34.931761 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:34.931711 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" podStartSLOduration=1.931696254 podStartE2EDuration="1.931696254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:34.929908188 +0000 UTC m=+814.704758035" watchObservedRunningTime="2026-11-03 21:59:34.931696254 +0000 UTC m=+814.706546100" Nov 03 21:59:35.147182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.147156 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:35.150512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.150491 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:35.190525 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190493 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\") pod \"568ba166-ca39-459f-94d8-b0ded051adc9\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " Nov 03 21:59:35.190680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190547 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\") pod \"568ba166-ca39-459f-94d8-b0ded051adc9\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " Nov 03 21:59:35.190680 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190639 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\" (UniqueName: \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\") pod \"568ba166-ca39-459f-94d8-b0ded051adc9\" (UID: \"568ba166-ca39-459f-94d8-b0ded051adc9\") " Nov 03 21:59:35.190819 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190691 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\" (UniqueName: \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\") pod \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " Nov 03 21:59:35.190880 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190847 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\") pod \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " Nov 03 21:59:35.190940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190880 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\") pod \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\" (UID: \"3224cbfb-42f0-45bd-a293-becc7d2cb02e\") " Nov 03 21:59:35.190940 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.190905 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca" pod "568ba166-ca39-459f-94d8-b0ded051adc9" (UID: "568ba166-ca39-459f-94d8-b0ded051adc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:35.191347 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.191310 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.191657 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.191597 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca" pod "3224cbfb-42f0-45bd-a293-becc7d2cb02e" (UID: "3224cbfb-42f0-45bd-a293-becc7d2cb02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:35.193266 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.193236 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn" pod "3224cbfb-42f0-45bd-a293-becc7d2cb02e" (UID: "3224cbfb-42f0-45bd-a293-becc7d2cb02e"). InnerVolumeSpecName "kube-api-access-9pcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:35.193436 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.193416 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca" pod "568ba166-ca39-459f-94d8-b0ded051adc9" (UID: "568ba166-ca39-459f-94d8-b0ded051adc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:35.193647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.193451 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca" pod "3224cbfb-42f0-45bd-a293-becc7d2cb02e" (UID: "3224cbfb-42f0-45bd-a293-becc7d2cb02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:35.193725 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.193702 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd" pod "568ba166-ca39-459f-94d8-b0ded051adc9" (UID: "568ba166-ca39-459f-94d8-b0ded051adc9"). InnerVolumeSpecName "kube-api-access-wh8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:35.292068 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.292038 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.292068 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.292064 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224cbfb-42f0-45bd-a293-becc7d2cb02e-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.292068 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.292074 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568ba166-ca39-459f-94d8-b0ded051adc9-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.292277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.292082 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8bd\" (UniqueName: \"kubernetes.io/projected/568ba166-ca39-459f-94d8-b0ded051adc9-kube-api-access-wh8bd\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.292277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.292092 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcsn\" (UniqueName: \"kubernetes.io/projected/3224cbfb-42f0-45bd-a293-becc7d2cb02e-kube-api-access-9pcsn\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:35.348780 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.348707 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69"] Nov 03 21:59:35.349185 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349172 5973 cpu_manager.go:414] "containerMap: removing container" podUID="568ba166-ca39-459f-94d8-b0ded051adc9" containerName="revoke-keys" Nov 03 21:59:35.349233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349186 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568ba166-ca39-459f-94d8-b0ded051adc9" containerName="revoke-keys" Nov 03 21:59:35.349233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349200 5973 cpu_manager.go:414] "containerMap: removing container" podUID="3224cbfb-42f0-45bd-a293-becc7d2cb02e" containerName="revoke-keys" Nov 03 21:59:35.349233 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349207 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3224cbfb-42f0-45bd-a293-becc7d2cb02e" containerName="revoke-keys" Nov 03 21:59:35.349317 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349301 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="568ba166-ca39-459f-94d8-b0ded051adc9" containerName="revoke-keys" Nov 03 21:59:35.349317 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.349310 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="3224cbfb-42f0-45bd-a293-becc7d2cb02e" containerName="revoke-keys" Nov 03 21:59:35.353346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.353315 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.356117 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.356097 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949"] Nov 03 21:59:35.359835 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.359811 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.362645 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.362626 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69"] Nov 03 21:59:35.369874 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.369845 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949"] Nov 03 21:59:35.393222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393199 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.393329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393247 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.393329 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393266 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.393433 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393404 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.393551 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393529 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.393654 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.393641 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjcw\" (UniqueName: \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.410616 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.410591 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4"] Nov 03 21:59:35.415320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.415295 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.422840 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.422657 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4"] Nov 03 21:59:35.494391 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494362 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjcw\" (UniqueName: \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.494589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494401 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95nkb\" (UniqueName: \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.494589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494446 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.494589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494539 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.494739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494594 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.494739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494623 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.494739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494663 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.494739 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494695 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.494937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.494794 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.495376 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.495354 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.495532 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.495377 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.497002 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.496977 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.497090 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.497040 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.507081 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.507054 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\") pod \"maas-api-revoke-sub-9e1a97ac-jp949\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.507277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.507261 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjcw\" (UniqueName: \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\") pod \"maas-api-revoke-sub-fe429fe1-p5r69\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.595424 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.595394 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95nkb\" (UniqueName: \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.595543 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.595485 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.595543 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.595516 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.596093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.596076 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.597836 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.597811 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.607868 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.607810 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95nkb\" (UniqueName: \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\") pod \"maas-api-revoke-sub-113d05e7-bq4s4\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.678641 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.678618 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:35.681754 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.681736 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:35.716262 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.716140 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:35.732310 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.732282 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:35.914859 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.914829 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" Nov 03 21:59:35.915000 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.914862 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv" event={"ID":"568ba166-ca39-459f-94d8-b0ded051adc9","Type":"ContainerDied","Data":"d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2"} Nov 03 21:59:35.915000 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.914910 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16d8eabea0ddd74c3b37d84dc6c2e4d0d92004804d20c080d8d209aa17520d2" Nov 03 21:59:35.916266 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.916243 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" Nov 03 21:59:35.916437 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.916266 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9" event={"ID":"3224cbfb-42f0-45bd-a293-becc7d2cb02e","Type":"ContainerDied","Data":"b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99"} Nov 03 21:59:35.916437 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.916307 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2df62b49d8317a8b16252506f43a0555615f6c63239ba3419b6da1f64e33b99" Nov 03 21:59:35.918020 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.917995 5973 generic.go:356] "Generic (PLEG): container finished" podID="e5185f19-313f-414f-b642-c9525ae8adfa" containerID="92add4f36332dfc62b666f82eda49d4051d4be64d26bc8d7cd6b51d9ddd03af6" exitCode=0 Nov 03 21:59:35.918127 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.918030 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" event={"ID":"e5185f19-313f-414f-b642-c9525ae8adfa","Type":"ContainerDied","Data":"92add4f36332dfc62b666f82eda49d4051d4be64d26bc8d7cd6b51d9ddd03af6"} Nov 03 21:59:35.996489 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:35.996379 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949"] Nov 03 21:59:35.998509 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:35.998440 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6906641a_8aa0_43f1_a26d_75cef444a128.slice/crio-35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1 WatchSource:0}: Error finding container 35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1: Status 404 returned error can't find the container with id 35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1 Nov 03 21:59:36.000714 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.000693 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69"] Nov 03 21:59:36.003651 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:36.003630 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4922b904_9dbc_4b27_bb51_741ef7cbae57.slice/crio-4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74 WatchSource:0}: Error finding container 4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74: Status 404 returned error can't find the container with id 4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74 Nov 03 21:59:36.039010 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.038987 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4"] Nov 03 21:59:36.042096 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:36.042071 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e91c153_32da_4427_9ea7_633dba8c415b.slice/crio-7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89 WatchSource:0}: Error finding container 7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89: Status 404 returned error can't find the container with id 7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89 Nov 03 21:59:36.529714 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.529597 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:36.925957 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.925929 5973 generic.go:356] "Generic (PLEG): container finished" podID="4922b904-9dbc-4b27-bb51-741ef7cbae57" containerID="55874fc71b8ac00014b27a06304ad9b27c0be21ed28d93adf33bfccc23cd1ab7" exitCode=0 Nov 03 21:59:36.926347 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.926011 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" event={"ID":"4922b904-9dbc-4b27-bb51-741ef7cbae57","Type":"ContainerDied","Data":"55874fc71b8ac00014b27a06304ad9b27c0be21ed28d93adf33bfccc23cd1ab7"} Nov 03 21:59:36.926347 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.926060 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" event={"ID":"4922b904-9dbc-4b27-bb51-741ef7cbae57","Type":"ContainerStarted","Data":"4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74"} Nov 03 21:59:36.927840 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.927808 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" event={"ID":"6906641a-8aa0-43f1-a26d-75cef444a128","Type":"ContainerStarted","Data":"7793ac33a5b642fdbedb7a94c73ec2dabb1e2e843915b021e33c9f3eb124bd98"} Nov 03 21:59:36.927941 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.927862 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" event={"ID":"6906641a-8aa0-43f1-a26d-75cef444a128","Type":"ContainerStarted","Data":"35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1"} Nov 03 21:59:36.929396 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.929374 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" event={"ID":"8e91c153-32da-4427-9ea7-633dba8c415b","Type":"ContainerStarted","Data":"c1d8236055c72e1f8bc85adf9686a15765446f14dc968b2c5dc046769677425e"} Nov 03 21:59:36.929506 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:36.929413 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" event={"ID":"8e91c153-32da-4427-9ea7-633dba8c415b","Type":"ContainerStarted","Data":"7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89"} Nov 03 21:59:37.032122 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.032067 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" podStartSLOduration=2.032051808 podStartE2EDuration="2.032051808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:37.030908581 +0000 UTC m=+816.805758429" watchObservedRunningTime="2026-11-03 21:59:37.032051808 +0000 UTC m=+816.806901691" Nov 03 21:59:37.052142 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.052094 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" podStartSLOduration=2.05207953 podStartE2EDuration="2.05207953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:37.048234187 +0000 UTC m=+816.823084035" watchObservedRunningTime="2026-11-03 21:59:37.05207953 +0000 UTC m=+816.826929547" Nov 03 21:59:37.142298 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.142281 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:37.213446 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.213418 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\") pod \"e5185f19-313f-414f-b642-c9525ae8adfa\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " Nov 03 21:59:37.213606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.213523 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\" (UniqueName: \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\") pod \"e5185f19-313f-414f-b642-c9525ae8adfa\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " Nov 03 21:59:37.213651 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.213611 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\") pod \"e5185f19-313f-414f-b642-c9525ae8adfa\" (UID: \"e5185f19-313f-414f-b642-c9525ae8adfa\") " Nov 03 21:59:37.213769 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.213741 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca" pod "e5185f19-313f-414f-b642-c9525ae8adfa" (UID: "e5185f19-313f-414f-b642-c9525ae8adfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:37.214013 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.213993 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:37.215656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.215632 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb" pod "e5185f19-313f-414f-b642-c9525ae8adfa" (UID: "e5185f19-313f-414f-b642-c9525ae8adfa"). InnerVolumeSpecName "kube-api-access-69msb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:37.215755 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.215651 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca" pod "e5185f19-313f-414f-b642-c9525ae8adfa" (UID: "e5185f19-313f-414f-b642-c9525ae8adfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:37.315195 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.315163 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69msb\" (UniqueName: \"kubernetes.io/projected/e5185f19-313f-414f-b642-c9525ae8adfa-kube-api-access-69msb\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:37.315195 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.315192 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5185f19-313f-414f-b642-c9525ae8adfa-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:37.346251 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.346216 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:37.936989 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.936906 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" Nov 03 21:59:37.936989 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.936941 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d" event={"ID":"e5185f19-313f-414f-b642-c9525ae8adfa","Type":"ContainerDied","Data":"4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9"} Nov 03 21:59:37.937590 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.937003 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5e173c495961d61b71c0f29a3ea8dfcb7203ce6c2497a05de56a787295a9d9" Nov 03 21:59:37.938743 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.938721 5973 generic.go:356] "Generic (PLEG): container finished" podID="6906641a-8aa0-43f1-a26d-75cef444a128" containerID="7793ac33a5b642fdbedb7a94c73ec2dabb1e2e843915b021e33c9f3eb124bd98" exitCode=0 Nov 03 21:59:37.938891 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.938775 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" event={"ID":"6906641a-8aa0-43f1-a26d-75cef444a128","Type":"ContainerDied","Data":"7793ac33a5b642fdbedb7a94c73ec2dabb1e2e843915b021e33c9f3eb124bd98"} Nov 03 21:59:37.940394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.940371 5973 generic.go:356] "Generic (PLEG): container finished" podID="8e91c153-32da-4427-9ea7-633dba8c415b" containerID="c1d8236055c72e1f8bc85adf9686a15765446f14dc968b2c5dc046769677425e" exitCode=0 Nov 03 21:59:37.940505 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:37.940475 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" event={"ID":"8e91c153-32da-4427-9ea7-633dba8c415b","Type":"ContainerDied","Data":"c1d8236055c72e1f8bc85adf9686a15765446f14dc968b2c5dc046769677425e"} Nov 03 21:59:38.038800 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.038749 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:38.110890 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.110865 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:38.129625 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.129603 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\") pod \"4922b904-9dbc-4b27-bb51-741ef7cbae57\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " Nov 03 21:59:38.129738 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.129687 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\" (UniqueName: \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\") pod \"4922b904-9dbc-4b27-bb51-741ef7cbae57\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " Nov 03 21:59:38.129782 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.129745 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\") pod \"4922b904-9dbc-4b27-bb51-741ef7cbae57\" (UID: \"4922b904-9dbc-4b27-bb51-741ef7cbae57\") " Nov 03 21:59:38.129925 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.129904 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca" pod "4922b904-9dbc-4b27-bb51-741ef7cbae57" (UID: "4922b904-9dbc-4b27-bb51-741ef7cbae57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:38.130072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.130053 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:38.132135 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.132079 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw" pod "4922b904-9dbc-4b27-bb51-741ef7cbae57" (UID: "4922b904-9dbc-4b27-bb51-741ef7cbae57"). InnerVolumeSpecName "kube-api-access-hxjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:38.132297 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.132275 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca" pod "4922b904-9dbc-4b27-bb51-741ef7cbae57" (UID: "4922b904-9dbc-4b27-bb51-741ef7cbae57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:38.231076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.231004 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4922b904-9dbc-4b27-bb51-741ef7cbae57-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:38.231076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.231026 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjcw\" (UniqueName: \"kubernetes.io/projected/4922b904-9dbc-4b27-bb51-741ef7cbae57-kube-api-access-hxjcw\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:38.784199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.784160 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" containerID="cri-o://78187af9464932aa5d414006a7e33838828d5c7af911ad567e9ed165c3b52e14" gracePeriod=45 Nov 03 21:59:38.952238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.952192 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Nov 03 21:59:38.959507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.959356 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" Nov 03 21:59:38.959507 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.959426 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69" event={"ID":"4922b904-9dbc-4b27-bb51-741ef7cbae57","Type":"ContainerDied","Data":"4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74"} Nov 03 21:59:38.959678 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.959524 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a65f8744559d2e62e589739aba7ca5d817786a0915dbef7ee31d1171df90d74" Nov 03 21:59:38.962167 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.962142 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:38.964781 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.963904 5973 generic.go:356] "Generic (PLEG): container finished" podID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerID="78187af9464932aa5d414006a7e33838828d5c7af911ad567e9ed165c3b52e14" exitCode=0 Nov 03 21:59:38.964781 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:38.964602 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" event={"ID":"7630ca6a-44af-48f0-b81b-f2c671946c6e","Type":"ContainerDied","Data":"78187af9464932aa5d414006a7e33838828d5c7af911ad567e9ed165c3b52e14"} Nov 03 21:59:39.104644 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.104276 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:59:39.141552 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141516 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.141690 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141637 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.141690 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141684 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.141805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141707 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.141805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141738 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.141913 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.141848 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\" (UniqueName: \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\") pod \"7630ca6a-44af-48f0-b81b-f2c671946c6e\" (UID: \"7630ca6a-44af-48f0-b81b-f2c671946c6e\") " Nov 03 21:59:39.142295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.142247 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:39.142295 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.142261 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:39.142480 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.142278 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:39.145320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.145281 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:39.145320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.145297 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "kube-api-access-xtc8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:39.145764 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.145726 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm" pod "7630ca6a-44af-48f0-b81b-f2c671946c6e" (UID: "7630ca6a-44af-48f0-b81b-f2c671946c6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:39.162564 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.162285 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:39.165110 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.165083 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:39.242989 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.242959 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\") pod \"8e91c153-32da-4427-9ea7-633dba8c415b\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " Nov 03 21:59:39.243143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243076 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\") pod \"6906641a-8aa0-43f1-a26d-75cef444a128\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " Nov 03 21:59:39.243143 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243123 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\") pod \"8e91c153-32da-4427-9ea7-633dba8c415b\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " Nov 03 21:59:39.243216 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243155 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\") pod \"6906641a-8aa0-43f1-a26d-75cef444a128\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " Nov 03 21:59:39.243216 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243196 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\") pod \"6906641a-8aa0-43f1-a26d-75cef444a128\" (UID: \"6906641a-8aa0-43f1-a26d-75cef444a128\") " Nov 03 21:59:39.243306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243263 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\" (UniqueName: \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\") pod \"8e91c153-32da-4427-9ea7-633dba8c415b\" (UID: \"8e91c153-32da-4427-9ea7-633dba8c415b\") " Nov 03 21:59:39.243502 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243469 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca" pod "6906641a-8aa0-43f1-a26d-75cef444a128" (UID: "6906641a-8aa0-43f1-a26d-75cef444a128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:39.243598 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243510 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca" pod "8e91c153-32da-4427-9ea7-633dba8c415b" (UID: "8e91c153-32da-4427-9ea7-633dba8c415b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:39.243776 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243758 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtc8w\" (UniqueName: \"kubernetes.io/projected/7630ca6a-44af-48f0-b81b-f2c671946c6e-kube-api-access-xtc8w\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243780 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243789 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243800 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243835 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243848 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243858 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7630ca6a-44af-48f0-b81b-f2c671946c6e-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.244012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.243869 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7630ca6a-44af-48f0-b81b-f2c671946c6e-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.245548 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.245527 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca" pod "6906641a-8aa0-43f1-a26d-75cef444a128" (UID: "6906641a-8aa0-43f1-a26d-75cef444a128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:39.245679 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.245583 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca" pod "8e91c153-32da-4427-9ea7-633dba8c415b" (UID: "8e91c153-32da-4427-9ea7-633dba8c415b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:39.245963 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.245947 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2" pod "6906641a-8aa0-43f1-a26d-75cef444a128" (UID: "6906641a-8aa0-43f1-a26d-75cef444a128"). InnerVolumeSpecName "kube-api-access-x46r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:39.246024 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.245969 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb" pod "8e91c153-32da-4427-9ea7-633dba8c415b" (UID: "8e91c153-32da-4427-9ea7-633dba8c415b"). InnerVolumeSpecName "kube-api-access-95nkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:39.345497 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.345441 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91c153-32da-4427-9ea7-633dba8c415b-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.345589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.345495 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/6906641a-8aa0-43f1-a26d-75cef444a128-kube-api-access-x46r2\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.345589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.345508 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6906641a-8aa0-43f1-a26d-75cef444a128-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.345589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.345517 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95nkb\" (UniqueName: \"kubernetes.io/projected/8e91c153-32da-4427-9ea7-633dba8c415b-kube-api-access-95nkb\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:39.658485 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.658415 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:39.972199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.972108 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" Nov 03 21:59:39.972199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.972124 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8" event={"ID":"7630ca6a-44af-48f0-b81b-f2c671946c6e","Type":"ContainerDied","Data":"f314444e970de26b7c4569bdc8facf3683e9757c56a61d7bab335f3df4315f98"} Nov 03 21:59:39.972199 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.972173 5973 scope.go:122] "RemoveContainer" containerID="78187af9464932aa5d414006a7e33838828d5c7af911ad567e9ed165c3b52e14" Nov 03 21:59:39.975196 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.975177 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" Nov 03 21:59:39.975736 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.975188 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949" event={"ID":"6906641a-8aa0-43f1-a26d-75cef444a128","Type":"ContainerDied","Data":"35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1"} Nov 03 21:59:39.975736 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.975691 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c26497ca5b2cc106232b8d7df6faae52e2cd843a3e69efdd573ea8504dc0f1" Nov 03 21:59:39.977234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.977210 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" event={"ID":"8e91c153-32da-4427-9ea7-633dba8c415b","Type":"ContainerDied","Data":"7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89"} Nov 03 21:59:39.977350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.977256 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2a01138095bac04dd382ee84556bd2bee7622e1d8815b2eba1c5f8ffb02d89" Nov 03 21:59:39.977350 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:39.977311 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4" Nov 03 21:59:40.000871 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.000846 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:59:40.006353 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.006333 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f4fb1b/simulator-w1-f4fb1b-kserve-99d675b87-c94h8"] Nov 03 21:59:40.037582 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.037559 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5"] Nov 03 21:59:40.038066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038050 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4922b904-9dbc-4b27-bb51-741ef7cbae57" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038068 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4922b904-9dbc-4b27-bb51-741ef7cbae57" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038086 5973 cpu_manager.go:414] "containerMap: removing container" podUID="e5185f19-313f-414f-b642-c9525ae8adfa" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038091 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5185f19-313f-414f-b642-c9525ae8adfa" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038097 5973 cpu_manager.go:414] "containerMap: removing container" podUID="8e91c153-32da-4427-9ea7-633dba8c415b" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038102 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e91c153-32da-4427-9ea7-633dba8c415b" containerName="revoke-keys" Nov 03 21:59:40.038124 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038132 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6906641a-8aa0-43f1-a26d-75cef444a128" containerName="revoke-keys" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038137 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6906641a-8aa0-43f1-a26d-75cef444a128" containerName="revoke-keys" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038144 5973 cpu_manager.go:414] "containerMap: removing container" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038149 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038221 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6906641a-8aa0-43f1-a26d-75cef444a128" containerName="revoke-keys" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038229 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" containerName="main" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038236 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="4922b904-9dbc-4b27-bb51-741ef7cbae57" containerName="revoke-keys" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038243 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5185f19-313f-414f-b642-c9525ae8adfa" containerName="revoke-keys" Nov 03 21:59:40.038306 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.038248 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e91c153-32da-4427-9ea7-633dba8c415b" containerName="revoke-keys" Nov 03 21:59:40.042889 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.042874 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.047131 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.047098 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:59:40.047240 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.047218 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:59:40.047365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.047301 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 21:59:40.047365 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.047338 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:59:40.052145 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.052126 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5"] Nov 03 21:59:40.109130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.109092 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:40.157777 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.157747 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.158040 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.158008 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdkh\" (UniqueName: \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.158112 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.158056 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.259553 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.259417 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdkh\" (UniqueName: \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.259553 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.259490 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.259772 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.259571 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.260341 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.260312 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.262317 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.262291 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.271851 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.271834 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdkh\" (UniqueName: \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\") pod \"maas-api-revoke-sub-03d351fb-4hrh5\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.309747 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.309715 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm"] Nov 03 21:59:40.313803 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.313783 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.323489 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.323451 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm"] Nov 03 21:59:40.356242 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.356212 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7630ca6a-44af-48f0-b81b-f2c671946c6e" path="/var/lib/kubelet/pods/7630ca6a-44af-48f0-b81b-f2c671946c6e/volumes" Nov 03 21:59:40.357595 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.357578 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:40.360741 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.360716 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.360833 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.360818 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.360988 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.360973 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mptt\" (UniqueName: \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.462103 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.462066 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mptt\" (UniqueName: \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.462264 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.462125 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.462264 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.462199 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.462846 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.462821 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.464671 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.464650 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.474258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.474239 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mptt\" (UniqueName: \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.544314 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.544221 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:40.630610 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.630582 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:40.636130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.636110 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5"] Nov 03 21:59:40.636999 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:40.636974 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299392bc_7989_43ed_93cb_24829eb17e23.slice/crio-79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382 WatchSource:0}: Error finding container 79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382: Status 404 returned error can't find the container with id 79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382 Nov 03 21:59:40.654677 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.654609 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerName="main" containerID="cri-o://a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351" gracePeriod=45 Nov 03 21:59:40.657527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.657393 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" podUID="ace72d8d-5412-43b6-a17c-f3158ef31910" containerName="main" containerID="cri-o://ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7" gracePeriod=45 Nov 03 21:59:40.808900 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.808862 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Nov 03 21:59:40.925788 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.925758 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm"] Nov 03 21:59:40.930516 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:40.930480 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab378c7_6261_4468_9c6d_cf62cafbc4fb.slice/crio-1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179 WatchSource:0}: Error finding container 1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179: Status 404 returned error can't find the container with id 1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179 Nov 03 21:59:40.933030 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.932976 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:59:40.968172 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968146 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968280 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968204 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968280 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968255 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968409 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968391 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\" (UniqueName: \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968480 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968435 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968533 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968482 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:40.968867 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968665 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\") pod \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\" (UID: \"1c549dd8-99c7-4adf-89eb-8d38dc032d8f\") " Nov 03 21:59:40.968867 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.968698 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:40.969167 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.969086 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:40.969337 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.969311 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:40.969389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.969342 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:40.969389 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.969357 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:40.971191 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.971170 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:40.971375 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.971354 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "kube-api-access-kcsvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:40.972562 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.972541 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm" pod "1c549dd8-99c7-4adf-89eb-8d38dc032d8f" (UID: "1c549dd8-99c7-4adf-89eb-8d38dc032d8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:40.982975 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.982951 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:59:40.986046 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.986020 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" event={"ID":"0ab378c7-6261-4468-9c6d-cf62cafbc4fb","Type":"ContainerStarted","Data":"1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179"} Nov 03 21:59:40.987281 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.987259 5973 generic.go:356] "Generic (PLEG): container finished" podID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerID="a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351" exitCode=0 Nov 03 21:59:40.987362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.987286 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" event={"ID":"1c549dd8-99c7-4adf-89eb-8d38dc032d8f","Type":"ContainerDied","Data":"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351"} Nov 03 21:59:40.987362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.987305 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" Nov 03 21:59:40.987362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.987326 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj" event={"ID":"1c549dd8-99c7-4adf-89eb-8d38dc032d8f","Type":"ContainerDied","Data":"74488b5494237a45a7a7e8d618eb3ec38579afbe55a1efff0b72529986f4b13d"} Nov 03 21:59:40.987362 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.987347 5973 scope.go:122] "RemoveContainer" containerID="a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351" Nov 03 21:59:40.989284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.989167 5973 generic.go:356] "Generic (PLEG): container finished" podID="ace72d8d-5412-43b6-a17c-f3158ef31910" containerID="ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7" exitCode=0 Nov 03 21:59:40.989284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.989202 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" event={"ID":"ace72d8d-5412-43b6-a17c-f3158ef31910","Type":"ContainerDied","Data":"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7"} Nov 03 21:59:40.989284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.989227 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" Nov 03 21:59:40.989284 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.989257 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr" event={"ID":"ace72d8d-5412-43b6-a17c-f3158ef31910","Type":"ContainerDied","Data":"1d1ae42007a081943294397cb947342abab0b457da193929e6fa3e30a8cb41b9"} Nov 03 21:59:40.990977 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:40.990945 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" event={"ID":"299392bc-7989-43ed-93cb-24829eb17e23","Type":"ContainerStarted","Data":"79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382"} Nov 03 21:59:41.000276 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.000260 5973 scope.go:122] "RemoveContainer" containerID="a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351" Nov 03 21:59:41.000572 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:41.000552 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351\": container with ID starting with a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351 not found: ID does not exist" containerID="a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351" Nov 03 21:59:41.000667 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.000578 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351"} err="failed to get container status \"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351\": rpc error: code = NotFound desc = could not find container \"a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351\": container with ID starting with a83d9a39c2e75723f57778b335d18b4044701ec591a3e9998e12dc641158f351 not found: ID does not exist" Nov 03 21:59:41.000667 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.000595 5973 scope.go:122] "RemoveContainer" containerID="ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7" Nov 03 21:59:41.016745 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.016658 5973 scope.go:122] "RemoveContainer" containerID="ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7" Nov 03 21:59:41.016981 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:41.016962 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7\": container with ID starting with ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7 not found: ID does not exist" containerID="ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7" Nov 03 21:59:41.017023 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.016988 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7"} err="failed to get container status \"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7\": rpc error: code = NotFound desc = could not find container \"ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7\": container with ID starting with ea29c65ebc404b72786d4f2fc0da2fd06f4ace5e1d5c3cae49001441f73e64a7 not found: ID does not exist" Nov 03 21:59:41.028129 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.028103 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:59:41.032750 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.032728 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/premium-w6-8af3f3-kserve-85f9df9b89-6xskj"] Nov 03 21:59:41.070606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070586 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\" (UniqueName: \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.070927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070621 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.070927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070666 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\" (UniqueName: \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.070927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070717 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.070927 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070818 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.071170 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.070965 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\") pod \"ace72d8d-5412-43b6-a17c-f3158ef31910\" (UID: \"ace72d8d-5412-43b6-a17c-f3158ef31910\") " Nov 03 21:59:41.071170 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071039 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:41.071170 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071115 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:41.071608 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071537 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:41.071691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071660 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcsvn\" (UniqueName: \"kubernetes.io/projected/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-kube-api-access-kcsvn\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.071691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071683 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.071829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071698 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.071829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071715 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.071829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071728 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.071829 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.071744 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c549dd8-99c7-4adf-89eb-8d38dc032d8f-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.072954 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.072930 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:41.073091 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.073073 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:41.073263 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.073245 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c" pod "ace72d8d-5412-43b6-a17c-f3158ef31910" (UID: "ace72d8d-5412-43b6-a17c-f3158ef31910"). InnerVolumeSpecName "kube-api-access-ml85c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:41.173411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.173376 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace72d8d-5412-43b6-a17c-f3158ef31910-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.173411 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.173407 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace72d8d-5412-43b6-a17c-f3158ef31910-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.173659 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.173421 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml85c\" (UniqueName: \"kubernetes.io/projected/ace72d8d-5412-43b6-a17c-f3158ef31910-kube-api-access-ml85c\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:41.206012 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.205986 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:41.316165 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.316058 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:59:41.319782 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.319755 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8af3f3/simulator-w6-8af3f3-kserve-568b69889b-m4twr"] Nov 03 21:59:41.495499 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.495442 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:41.629850 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.629765 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws"] Nov 03 21:59:41.632648 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.632628 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c0caef-xwxws"] Nov 03 21:59:41.831066 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.831034 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" containerID="cri-o://f7642d3a5eef585717418f0db55d2fe76c1f0d87607315f566bf49edbac18d7a" gracePeriod=45 Nov 03 21:59:41.999235 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.999203 5973 generic.go:356] "Generic (PLEG): container finished" podID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerID="f7642d3a5eef585717418f0db55d2fe76c1f0d87607315f566bf49edbac18d7a" exitCode=0 Nov 03 21:59:41.999703 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:41.999278 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" event={"ID":"8686da36-b720-489e-bbfa-d3d3e60ca278","Type":"ContainerDied","Data":"f7642d3a5eef585717418f0db55d2fe76c1f0d87607315f566bf49edbac18d7a"} Nov 03 21:59:42.001125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.001092 5973 generic.go:356] "Generic (PLEG): container finished" podID="0ab378c7-6261-4468-9c6d-cf62cafbc4fb" containerID="79ab197eefcfdd75b09dbaaf24339d48c6172498dc740a169b99e349f361f7bc" exitCode=0 Nov 03 21:59:42.001239 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.001173 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" event={"ID":"0ab378c7-6261-4468-9c6d-cf62cafbc4fb","Type":"ContainerDied","Data":"79ab197eefcfdd75b09dbaaf24339d48c6172498dc740a169b99e349f361f7bc"} Nov 03 21:59:42.007709 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.005808 5973 generic.go:356] "Generic (PLEG): container finished" podID="299392bc-7989-43ed-93cb-24829eb17e23" containerID="4ab1815edca16b7afb446d42836102e24afe32ec85db2c1bd78d2d1d8e0b6b36" exitCode=0 Nov 03 21:59:42.007709 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.005844 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" event={"ID":"299392bc-7989-43ed-93cb-24829eb17e23","Type":"ContainerDied","Data":"4ab1815edca16b7afb446d42836102e24afe32ec85db2c1bd78d2d1d8e0b6b36"} Nov 03 21:59:42.096520 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.096495 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:59:42.192157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192130 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192319 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192172 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192319 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192202 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192319 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192222 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\" (UniqueName: \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192316 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192353 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\" (UniqueName: \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\") pod \"8686da36-b720-489e-bbfa-d3d3e60ca278\" (UID: \"8686da36-b720-489e-bbfa-d3d3e60ca278\") " Nov 03 21:59:42.192601 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192504 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:42.192601 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192517 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:42.192699 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192656 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:42.192823 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192805 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.192867 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192826 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.192867 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.192835 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.194641 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.194621 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:42.194935 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.194917 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:42.194998 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.194957 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5" pod "8686da36-b720-489e-bbfa-d3d3e60ca278" (UID: "8686da36-b720-489e-bbfa-d3d3e60ca278"). InnerVolumeSpecName "kube-api-access-xf9p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:42.294031 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.293999 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8686da36-b720-489e-bbfa-d3d3e60ca278-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.294031 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.294030 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9p5\" (UniqueName: \"kubernetes.io/projected/8686da36-b720-489e-bbfa-d3d3e60ca278-kube-api-access-xf9p5\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.294222 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.294043 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8686da36-b720-489e-bbfa-d3d3e60ca278-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:42.345612 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.345581 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" path="/var/lib/kubelet/pods/1c549dd8-99c7-4adf-89eb-8d38dc032d8f/volumes" Nov 03 21:59:42.345995 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.345982 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace72d8d-5412-43b6-a17c-f3158ef31910" path="/var/lib/kubelet/pods/ace72d8d-5412-43b6-a17c-f3158ef31910/volumes" Nov 03 21:59:42.346333 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:42.346321 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7" path="/var/lib/kubelet/pods/d05e0a2e-a84a-4a7c-b22d-df3cd894e6e7/volumes" Nov 03 21:59:43.013895 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.013848 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" Nov 03 21:59:43.013895 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.013863 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp" event={"ID":"8686da36-b720-489e-bbfa-d3d3e60ca278","Type":"ContainerDied","Data":"9dc88c9873f683e8549f34d4dee13417ae0751e258dae9f5e0300faf013cf1cc"} Nov 03 21:59:43.014400 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.013921 5973 scope.go:122] "RemoveContainer" containerID="f7642d3a5eef585717418f0db55d2fe76c1f0d87607315f566bf49edbac18d7a" Nov 03 21:59:43.034711 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.034673 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:59:43.041805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.041776 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-85b4ff/simulator-w5-85b4ff-kserve-fcfbfd779-vw7jp"] Nov 03 21:59:43.166209 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.166186 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:43.206340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.206320 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:43.237062 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.237029 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:43.305156 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305069 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\" (UniqueName: \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\") pod \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " Nov 03 21:59:43.305320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305168 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\") pod \"299392bc-7989-43ed-93cb-24829eb17e23\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " Nov 03 21:59:43.305320 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305229 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\" (UniqueName: \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\") pod \"299392bc-7989-43ed-93cb-24829eb17e23\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " Nov 03 21:59:43.305449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305320 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\") pod \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " Nov 03 21:59:43.305449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305385 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\") pod \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\" (UID: \"0ab378c7-6261-4468-9c6d-cf62cafbc4fb\") " Nov 03 21:59:43.305449 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.305443 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\") pod \"299392bc-7989-43ed-93cb-24829eb17e23\" (UID: \"299392bc-7989-43ed-93cb-24829eb17e23\") " Nov 03 21:59:43.306358 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.306324 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca" pod "299392bc-7989-43ed-93cb-24829eb17e23" (UID: "299392bc-7989-43ed-93cb-24829eb17e23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:43.306749 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.306722 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca" pod "0ab378c7-6261-4468-9c6d-cf62cafbc4fb" (UID: "0ab378c7-6261-4468-9c6d-cf62cafbc4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:43.308758 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.308731 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca" pod "299392bc-7989-43ed-93cb-24829eb17e23" (UID: "299392bc-7989-43ed-93cb-24829eb17e23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:43.308870 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.308754 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt" pod "0ab378c7-6261-4468-9c6d-cf62cafbc4fb" (UID: "0ab378c7-6261-4468-9c6d-cf62cafbc4fb"). InnerVolumeSpecName "kube-api-access-4mptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:43.309074 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.309057 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh" pod "299392bc-7989-43ed-93cb-24829eb17e23" (UID: "299392bc-7989-43ed-93cb-24829eb17e23"). InnerVolumeSpecName "kube-api-access-qvdkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:43.310259 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.310234 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca" pod "0ab378c7-6261-4468-9c6d-cf62cafbc4fb" (UID: "0ab378c7-6261-4468-9c6d-cf62cafbc4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:43.344867 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.344832 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerName="main" containerID="cri-o://461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b" gracePeriod=45 Nov 03 21:59:43.403065 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:43.403024 5973 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6c8e21_db51_4cbf_81f2_868cbc1eff2f.slice/crio-conmon-461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6c8e21_db51_4cbf_81f2_868cbc1eff2f.slice/crio-461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:59:43.406804 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406779 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mptt\" (UniqueName: \"kubernetes.io/projected/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-kube-api-access-4mptt\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.406918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406807 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.406918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406821 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvdkh\" (UniqueName: \"kubernetes.io/projected/299392bc-7989-43ed-93cb-24829eb17e23-kube-api-access-qvdkh\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.406918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406834 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.406918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406847 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab378c7-6261-4468-9c6d-cf62cafbc4fb-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.406918 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.406861 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299392bc-7989-43ed-93cb-24829eb17e23-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.425640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.425610 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:43.592537 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.585505 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78"] Nov 03 21:59:43.598528 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.596329 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:43.614477 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.604518 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-85b4ff-wjr78"] Nov 03 21:59:43.661286 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.661262 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:59:43.710346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710260 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.710346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710307 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.710346 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710335 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\" (UniqueName: \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.710652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710380 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.710652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710447 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.710652 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.710492 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\") pod \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\" (UID: \"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f\") " Nov 03 21:59:43.711304 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.711224 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:43.711523 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.711422 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:43.711589 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.711554 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:43.713629 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.713604 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "kube-api-access-9c6vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:43.713728 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.713685 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:43.713815 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.713788 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs" pod "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" (UID: "4b6c8e21-db51-4cbf-81f2-868cbc1eff2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:43.746520 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.746486 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:43.811653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811628 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.811653 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811653 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.811802 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811664 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.811802 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811673 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.811802 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811682 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c6vf\" (UniqueName: \"kubernetes.io/projected/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-kube-api-access-9c6vf\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:43.811802 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:43.811691 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:44.021688 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.021650 5973 generic.go:356] "Generic (PLEG): container finished" podID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerID="461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b" exitCode=0 Nov 03 21:59:44.022173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.021700 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" Nov 03 21:59:44.022173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.021739 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" event={"ID":"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f","Type":"ContainerDied","Data":"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b"} Nov 03 21:59:44.022173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.021790 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92" event={"ID":"4b6c8e21-db51-4cbf-81f2-868cbc1eff2f","Type":"ContainerDied","Data":"a44e6bd3c0f167268c733780058ab309c88ba31ec8c6c97c95f47c636e19fe9b"} Nov 03 21:59:44.022173 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.021814 5973 scope.go:122] "RemoveContainer" containerID="461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b" Nov 03 21:59:44.023272 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.023247 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" Nov 03 21:59:44.023383 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.023311 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5" event={"ID":"299392bc-7989-43ed-93cb-24829eb17e23","Type":"ContainerDied","Data":"79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382"} Nov 03 21:59:44.023383 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.023375 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79fbfc3e0c38ce5f1c24997ffccdfa646a958f85e641a7f3e4b16813ee55f382" Nov 03 21:59:44.026401 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.026376 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" event={"ID":"0ab378c7-6261-4468-9c6d-cf62cafbc4fb","Type":"ContainerDied","Data":"1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179"} Nov 03 21:59:44.026529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.026400 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm" Nov 03 21:59:44.026529 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.026435 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1306cbe5bc361b9aa51278eb8ae63bc52fc7697ca8bdde630a9be19d21ac5179" Nov 03 21:59:44.034368 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.034297 5973 scope.go:122] "RemoveContainer" containerID="461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b" Nov 03 21:59:44.034922 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:44.034900 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b\": container with ID starting with 461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b not found: ID does not exist" containerID="461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b" Nov 03 21:59:44.035007 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.034934 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b"} err="failed to get container status \"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b\": rpc error: code = NotFound desc = could not find container \"461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b\": container with ID starting with 461ff85c80bdb208cdf7ca85ff14021ecf5f832e65f11058f61aa46af6dc7d8b not found: ID does not exist" Nov 03 21:59:44.047030 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.047010 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:59:44.057809 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.057772 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-02e8ab/simulator-w3-02e8ab-kserve-64bbb6d6bb-9st92"] Nov 03 21:59:44.083611 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.083583 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.345946 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.345868 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" path="/var/lib/kubelet/pods/4b6c8e21-db51-4cbf-81f2-868cbc1eff2f/volumes" Nov 03 21:59:44.346270 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.346257 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5532faed-f1c4-4d1a-ac00-87e8125f7f3d" path="/var/lib/kubelet/pods/5532faed-f1c4-4d1a-ac00-87e8125f7f3d/volumes" Nov 03 21:59:44.346643 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.346630 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" path="/var/lib/kubelet/pods/8686da36-b720-489e-bbfa-d3d3e60ca278/volumes" Nov 03 21:59:44.402178 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:44.402129 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 21:59:44.521577 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.521539 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.655513 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.649575 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:59:44.683130 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.683094 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.837696 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.837667 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.988857 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:44.988767 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:45.194851 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:45.194823 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 21:59:45.353508 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:45.353402 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:45.514412 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:45.514372 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:45.908574 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:45.908525 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:46.750844 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:46.750809 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:47.390110 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.390075 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:47.523709 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.523673 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82"] Nov 03 21:59:47.526595 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.526568 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8af3f3-5gs82"] Nov 03 21:59:47.545972 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.545937 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:47.700724 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.700677 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:47.859949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:47.859907 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:48.099532 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:48.099423 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:48.328592 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:48.328528 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:48.345036 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:48.345014 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8557be75-c443-4a89-bfc7-c00e702eb0bb" path="/var/lib/kubelet/pods/8557be75-c443-4a89-bfc7-c00e702eb0bb/volumes" Nov 03 21:59:48.352819 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:48.352764 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 21:59:48.726310 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:48.726281 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:49.602483 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:49.600926 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:49.920394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:49.920363 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.069149 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.069123 5973 generic.go:356] "Generic (PLEG): container finished" podID="addd4340-d564-4812-9596-92e770fa4534" containerID="af74429d511493ed6f7a61d9e9504f7b07e649e96b475f37a7b9a0674eeb9f91" exitCode=0 Nov 03 21:59:50.069267 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.069193 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" event={"ID":"addd4340-d564-4812-9596-92e770fa4534","Type":"ContainerDied","Data":"af74429d511493ed6f7a61d9e9504f7b07e649e96b475f37a7b9a0674eeb9f91"} Nov 03 21:59:50.069267 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.069240 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" event={"ID":"addd4340-d564-4812-9596-92e770fa4534","Type":"ContainerDied","Data":"5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57"} Nov 03 21:59:50.069267 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.069263 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bbad83a3a4cf1b145b6bb91cb16b60527776b38a4634ebdffb2b03f008fda57" Nov 03 21:59:50.070071 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.070055 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:59:50.074575 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.074553 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.193292 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.193201 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:59:50.193493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.193441 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" containerID="cri-o://c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c" gracePeriod=30 Nov 03 21:59:50.200247 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.200226 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\") pod \"addd4340-d564-4812-9596-92e770fa4534\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " Nov 03 21:59:50.200384 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.200345 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\" (UniqueName: \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\") pod \"addd4340-d564-4812-9596-92e770fa4534\" (UID: \"addd4340-d564-4812-9596-92e770fa4534\") " Nov 03 21:59:50.200606 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.200584 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume" pod "addd4340-d564-4812-9596-92e770fa4534" (UID: "addd4340-d564-4812-9596-92e770fa4534"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:50.200748 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.200733 5973 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd4340-d564-4812-9596-92e770fa4534-plugins-config-volume\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:50.202400 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.202378 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz" pod "addd4340-d564-4812-9596-92e770fa4534" (UID: "addd4340-d564-4812-9596-92e770fa4534"). InnerVolumeSpecName "kube-api-access-n98cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:50.281601 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.281572 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.302028 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.302003 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98cz\" (UniqueName: \"kubernetes.io/projected/addd4340-d564-4812-9596-92e770fa4534-kube-api-access-n98cz\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:50.530198 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530108 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh"] Nov 03 21:59:50.530737 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530714 5973 cpu_manager.go:414] "containerMap: removing container" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530748 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530761 5973 cpu_manager.go:414] "containerMap: removing container" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530768 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530783 5973 cpu_manager.go:414] "containerMap: removing container" podUID="0ab378c7-6261-4468-9c6d-cf62cafbc4fb" containerName="revoke-keys" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530789 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab378c7-6261-4468-9c6d-cf62cafbc4fb" containerName="revoke-keys" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530798 5973 cpu_manager.go:414] "containerMap: removing container" podUID="ace72d8d-5412-43b6-a17c-f3158ef31910" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530802 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace72d8d-5412-43b6-a17c-f3158ef31910" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530816 5973 cpu_manager.go:414] "containerMap: removing container" podUID="299392bc-7989-43ed-93cb-24829eb17e23" containerName="revoke-keys" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530821 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299392bc-7989-43ed-93cb-24829eb17e23" containerName="revoke-keys" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530836 5973 cpu_manager.go:414] "containerMap: removing container" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530840 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530847 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerName="main" Nov 03 21:59:50.530878 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530851 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerName="main" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530922 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="8686da36-b720-489e-bbfa-d3d3e60ca278" containerName="main" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530929 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c549dd8-99c7-4adf-89eb-8d38dc032d8f" containerName="main" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530936 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab378c7-6261-4468-9c6d-cf62cafbc4fb" containerName="revoke-keys" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530942 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace72d8d-5412-43b6-a17c-f3158ef31910" containerName="main" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530949 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="299392bc-7989-43ed-93cb-24829eb17e23" containerName="revoke-keys" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530957 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="addd4340-d564-4812-9596-92e770fa4534" containerName="payload-processing" Nov 03 21:59:50.531277 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.530963 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6c8e21-db51-4cbf-81f2-868cbc1eff2f" containerName="main" Nov 03 21:59:50.534300 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.534284 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.541960 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.541941 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:59:50.542072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.541982 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:59:50.542547 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.542529 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 21:59:50.542647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.542607 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:59:50.545620 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.545598 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh"] Nov 03 21:59:50.548733 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.548703 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.605902 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.605868 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.606366 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.605928 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdtfh\" (UniqueName: \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.606366 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.606102 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.706958 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.706921 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.707157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.707081 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.707157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.707116 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdtfh\" (UniqueName: \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.707499 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.707442 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:50.707753 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.707730 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.709725 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.709702 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.723552 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.723525 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdtfh\" (UniqueName: \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\") pod \"maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.849757 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.849682 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:50.868614 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:50.868584 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:51.077109 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.077073 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk" Nov 03 21:59:51.095915 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.095893 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:59:51.101093 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.101036 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c0caef-8d76db98-zspvk"] Nov 03 21:59:51.128265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.128245 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh"] Nov 03 21:59:51.129242 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:51.129220 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d7f37b_404a_4dbb_8971_43201013b7ad.slice/crio-1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686 WatchSource:0}: Error finding container 1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686: Status 404 returned error can't find the container with id 1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686 Nov 03 21:59:51.641087 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.640979 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:51.802926 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.802876 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:51.950588 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:51.950552 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:52.055933 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.055904 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:59:52.056154 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.056128 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" containerID="cri-o://45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309" gracePeriod=30 Nov 03 21:59:52.094327 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.093684 5973 generic.go:356] "Generic (PLEG): container finished" podID="58d7f37b-404a-4dbb-8971-43201013b7ad" containerID="cdb074b3fa189cf687af6d62f63caa04323fa93a98dab49f7722311daa770f7a" exitCode=0 Nov 03 21:59:52.094327 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.093835 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" event={"ID":"58d7f37b-404a-4dbb-8971-43201013b7ad","Type":"ContainerDied","Data":"cdb074b3fa189cf687af6d62f63caa04323fa93a98dab49f7722311daa770f7a"} Nov 03 21:59:52.094327 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.093888 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" event={"ID":"58d7f37b-404a-4dbb-8971-43201013b7ad","Type":"ContainerStarted","Data":"1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686"} Nov 03 21:59:52.142700 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.142664 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:52.265252 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.265070 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw"] Nov 03 21:59:52.269822 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.269799 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.279371 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.279341 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw"] Nov 03 21:59:52.301775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.301750 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:52.329500 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.329443 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.329627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.329510 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.329627 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.329546 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4s9\" (UniqueName: \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.347324 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.347297 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addd4340-d564-4812-9596-92e770fa4534" path="/var/lib/kubelet/pods/addd4340-d564-4812-9596-92e770fa4534/volumes" Nov 03 21:59:52.431226 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.431201 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.431226 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.431227 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.431390 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.431256 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4s9\" (UniqueName: \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.436486 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.432274 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.436486 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.434716 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.443557 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.443534 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4s9\" (UniqueName: \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\") pod \"maas-api-revoke-sub-e4bac426-x4svw\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.599347 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.598445 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:52.607536 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.607501 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:52.886084 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.886042 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw"] Nov 03 21:59:52.886630 ip-10-0-131-89 kubenswrapper[5973]: W1103 21:59:52.886606 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba4ca7f_bbd2_4280_8598_ef50d031b42e.slice/crio-3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97 WatchSource:0}: Error finding container 3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97: Status 404 returned error can't find the container with id 3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97 Nov 03 21:59:52.951148 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:52.951111 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:53.101637 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.101605 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" event={"ID":"fba4ca7f-bbd2-4280-8598-ef50d031b42e","Type":"ContainerStarted","Data":"3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97"} Nov 03 21:59:53.106518 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.106489 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:53.180626 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.180599 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:59:53.180795 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.180774 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" containerID="cri-o://8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77" gracePeriod=30 Nov 03 21:59:53.275415 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.275370 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:53.310792 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.310759 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:53.443323 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.443249 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\") pod \"58d7f37b-404a-4dbb-8971-43201013b7ad\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " Nov 03 21:59:53.443512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.443359 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\") pod \"58d7f37b-404a-4dbb-8971-43201013b7ad\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " Nov 03 21:59:53.443512 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.443382 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\" (UniqueName: \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\") pod \"58d7f37b-404a-4dbb-8971-43201013b7ad\" (UID: \"58d7f37b-404a-4dbb-8971-43201013b7ad\") " Nov 03 21:59:53.443640 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.443618 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca" pod "58d7f37b-404a-4dbb-8971-43201013b7ad" (UID: "58d7f37b-404a-4dbb-8971-43201013b7ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:53.443862 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.443836 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:53.445691 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.445665 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh" pod "58d7f37b-404a-4dbb-8971-43201013b7ad" (UID: "58d7f37b-404a-4dbb-8971-43201013b7ad"). InnerVolumeSpecName "kube-api-access-vdtfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:53.445790 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.445740 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca" pod "58d7f37b-404a-4dbb-8971-43201013b7ad" (UID: "58d7f37b-404a-4dbb-8971-43201013b7ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:53.545467 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.545435 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d7f37b-404a-4dbb-8971-43201013b7ad-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:53.545632 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.545493 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdtfh\" (UniqueName: \"kubernetes.io/projected/58d7f37b-404a-4dbb-8971-43201013b7ad-kube-api-access-vdtfh\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:53.564488 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:53.564431 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:54.110125 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.110052 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" Nov 03 21:59:54.110563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.110053 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh" event={"ID":"58d7f37b-404a-4dbb-8971-43201013b7ad","Type":"ContainerDied","Data":"1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686"} Nov 03 21:59:54.110563 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.110184 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac84287e801e5bd4c41792e5341b75994d31ce5d50ce1276b2917a574a93686" Nov 03 21:59:54.111817 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.111792 5973 generic.go:356] "Generic (PLEG): container finished" podID="fba4ca7f-bbd2-4280-8598-ef50d031b42e" containerID="4a2fc97e1f80defe9fe9cd875f1ad19e8fe8115adf769ccd22ee0fb8f6b2a3cd" exitCode=0 Nov 03 21:59:54.111933 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.111833 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" event={"ID":"fba4ca7f-bbd2-4280-8598-ef50d031b42e","Type":"ContainerDied","Data":"4a2fc97e1f80defe9fe9cd875f1ad19e8fe8115adf769ccd22ee0fb8f6b2a3cd"} Nov 03 21:59:54.334034 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:54.333999 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:55.251937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.251716 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:55.321950 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.321929 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:55.365839 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.365802 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 21:59:55.414487 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.413192 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:55.465949 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.465926 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\") pod \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " Nov 03 21:59:55.466079 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.466049 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\" (UniqueName: \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\") pod \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " Nov 03 21:59:55.466157 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.466102 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\") pod \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\" (UID: \"fba4ca7f-bbd2-4280-8598-ef50d031b42e\") " Nov 03 21:59:55.466282 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.466257 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca" pod "fba4ca7f-bbd2-4280-8598-ef50d031b42e" (UID: "fba4ca7f-bbd2-4280-8598-ef50d031b42e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:55.466651 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.466620 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.468283 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.468259 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca" pod "fba4ca7f-bbd2-4280-8598-ef50d031b42e" (UID: "fba4ca7f-bbd2-4280-8598-ef50d031b42e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:55.468376 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.468310 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9" pod "fba4ca7f-bbd2-4280-8598-ef50d031b42e" (UID: "fba4ca7f-bbd2-4280-8598-ef50d031b42e"). InnerVolumeSpecName "kube-api-access-tq4s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:55.473153 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.473133 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:59:55.567167 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567133 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567330 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567211 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567330 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567237 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567330 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567313 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567361 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567422 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567451 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567497 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:55.567534 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567504 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:55.567775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567579 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567633 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:55.567775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567651 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:55.567775 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567644 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\") pod \"b37b4367-bffb-407c-896e-0c2b463fafd4\" (UID: \"b37b4367-bffb-407c-896e-0c2b463fafd4\") " Nov 03 21:59:55.567951 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.567873 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:55.568137 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568100 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4s9\" (UniqueName: \"kubernetes.io/projected/fba4ca7f-bbd2-4280-8598-ef50d031b42e-kube-api-access-tq4s9\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568211 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568298 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba4ca7f-bbd2-4280-8598-ef50d031b42e-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568312 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37b4367-bffb-407c-896e-0c2b463fafd4-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568327 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568340 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.568382 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.568355 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.569706 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.569681 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:59:55.569842 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.569820 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "kube-api-access-s96cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:55.569895 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.569832 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:55.569895 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.569839 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token" pod "b37b4367-bffb-407c-896e-0c2b463fafd4" (UID: "b37b4367-bffb-407c-896e-0c2b463fafd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:55.615762 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.615688 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:55.639770 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.639707 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 21:59:55.639922 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.639896 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" containerID="cri-o://c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0" gracePeriod=30 Nov 03 21:59:55.669182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.669153 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.669182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.669180 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.669182 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.669190 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96cs\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-kube-api-access-s96cs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.669394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.669200 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37b4367-bffb-407c-896e-0c2b463fafd4-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:55.776056 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:55.775990 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:56.135119 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.135081 5973 generic.go:356] "Generic (PLEG): container finished" podID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerID="c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c" exitCode=0 Nov 03 21:59:56.135339 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.135198 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" Nov 03 21:59:56.135493 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.135173 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" event={"ID":"b37b4367-bffb-407c-896e-0c2b463fafd4","Type":"ContainerDied","Data":"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c"} Nov 03 21:59:56.135771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.135617 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr" event={"ID":"b37b4367-bffb-407c-896e-0c2b463fafd4","Type":"ContainerDied","Data":"450cd65ce81b6bc870de1842b9f02e2ad1d26ce4e0d171338e00039f77a79173"} Nov 03 21:59:56.135771 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.135708 5973 scope.go:122] "RemoveContainer" containerID="c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c" Nov 03 21:59:56.137391 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.137374 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" Nov 03 21:59:56.137489 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.137415 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw" event={"ID":"fba4ca7f-bbd2-4280-8598-ef50d031b42e","Type":"ContainerDied","Data":"3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97"} Nov 03 21:59:56.137539 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.137497 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cda529e505fa0429ef4bf3a5d4b3e16de0ba2d1ec15a07dff1ad17f06319d97" Nov 03 21:59:56.148527 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.148507 5973 scope.go:122] "RemoveContainer" containerID="c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c" Nov 03 21:59:56.148874 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:56.148850 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c\": container with ID starting with c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c not found: ID does not exist" containerID="c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c" Nov 03 21:59:56.148953 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.148886 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c"} err="failed to get container status \"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c\": rpc error: code = NotFound desc = could not find container \"c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c\": container with ID starting with c7c6d2317cb227a942f9743a5f8ad64f0845afaf6bd9bbdc2b755de1c6c7a44c not found: ID does not exist" Nov 03 21:59:56.164805 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.164783 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:59:56.169318 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.169298 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c0caef-openshift-default-58fbf8db4f-7kddr"] Nov 03 21:59:56.328647 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.328604 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:56.345340 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.345312 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" path="/var/lib/kubelet/pods/b37b4367-bffb-407c-896e-0c2b463fafd4/volumes" Nov 03 21:59:56.539799 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.539763 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:56.721231 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.721202 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 21:59:56.721466 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.721431 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" containerID="cri-o://43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379" gracePeriod=30 Nov 03 21:59:56.807072 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:56.806983 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:57.172864 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.172364 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:57.320204 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.320176 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:59:57.391332 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391260 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.391332 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391297 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.391746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391350 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.391746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391604 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.391746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391650 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:57.391746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391653 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.391746 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391721 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.392027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391763 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.392027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391795 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.392027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391856 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\") pod \"7290c8e9-990d-4032-96e3-501b933cdc9b\" (UID: \"7290c8e9-990d-4032-96e3-501b933cdc9b\") " Nov 03 21:59:57.392027 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.391996 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:57.392258 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392201 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:57.392501 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392382 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:57.392594 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392571 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.392660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392600 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.392660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392615 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.392660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392619 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:57.392660 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.392630 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.393731 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.393701 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:57.393862 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.393844 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:57.393937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.393919 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:59:57.393937 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.393926 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn" pod "7290c8e9-990d-4032-96e3-501b933cdc9b" (UID: "7290c8e9-990d-4032-96e3-501b933cdc9b"). InnerVolumeSpecName "kube-api-access-9k4qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:57.493360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.493331 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4qn\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-kube-api-access-9k4qn\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.493360 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.493357 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.493560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.493367 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7290c8e9-990d-4032-96e3-501b933cdc9b-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.493560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.493378 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:57.493560 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:57.493387 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7290c8e9-990d-4032-96e3-501b933cdc9b-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.130522 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.130490 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:58.153305 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.153279 5973 generic.go:356] "Generic (PLEG): container finished" podID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerID="45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309" exitCode=0 Nov 03 21:59:58.153428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.153309 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerDied","Data":"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309"} Nov 03 21:59:58.153428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.153332 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" Nov 03 21:59:58.153428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.153363 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q" event={"ID":"7290c8e9-990d-4032-96e3-501b933cdc9b","Type":"ContainerDied","Data":"ede6724b0f6be08c28394b7237ad888a074863c9ba90aaa0ca493f1c134ac915"} Nov 03 21:59:58.153428 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.153385 5973 scope.go:122] "RemoveContainer" containerID="45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309" Nov 03 21:59:58.164439 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.164425 5973 scope.go:122] "RemoveContainer" containerID="d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b" Nov 03 21:59:58.175734 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.175574 5973 scope.go:122] "RemoveContainer" containerID="45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309" Nov 03 21:59:58.175891 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:58.175864 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309\": container with ID starting with 45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309 not found: ID does not exist" containerID="45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309" Nov 03 21:59:58.175945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.175902 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309"} err="failed to get container status \"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309\": rpc error: code = NotFound desc = could not find container \"45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309\": container with ID starting with 45d04b3c8afc7105697c824ff1123bdd2b0bb14d44ed0e406fb11ebf0939a309 not found: ID does not exist" Nov 03 21:59:58.175945 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.175924 5973 scope.go:122] "RemoveContainer" containerID="d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b" Nov 03 21:59:58.176197 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:58.176179 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b\": container with ID starting with d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b not found: ID does not exist" containerID="d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b" Nov 03 21:59:58.176234 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.176203 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b"} err="failed to get container status \"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b\": rpc error: code = NotFound desc = could not find container \"d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b\": container with ID starting with d2c94598f64a5b2a6cb28b20c137361dd737d48ad51c226e3a25df13d3fed90b not found: ID does not exist" Nov 03 21:59:58.187316 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.187285 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:59:58.190140 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.190122 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-85b4ff-openshift-default-84c9ccb77-scb4q"] Nov 03 21:59:58.290227 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.290197 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:58.346783 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.346753 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" path="/var/lib/kubelet/pods/7290c8e9-990d-4032-96e3-501b933cdc9b/volumes" Nov 03 21:59:58.443419 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.443398 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:59:58.601319 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601290 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601331 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601363 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601417 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601490 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601446 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601581 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601635 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601627 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:58.601682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601658 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:58.601682 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601674 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601936 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601748 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\") pod \"b5023934-6d80-4502-8b04-ee63751ed4f7\" (UID: \"b5023934-6d80-4502-8b04-ee63751ed4f7\") " Nov 03 21:59:58.601936 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601778 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:58.601936 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601877 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:58.602076 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.601975 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:58.602179 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.602160 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5023934-6d80-4502-8b04-ee63751ed4f7-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.602238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.602186 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.602238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.602202 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.602238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.602216 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.602238 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.602230 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.603808 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.603784 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "kube-api-access-b292s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:58.603952 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.603928 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:59:58.604177 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.604164 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:59:58.604265 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.604243 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token" pod "b5023934-6d80-4502-8b04-ee63751ed4f7" (UID: "b5023934-6d80-4502-8b04-ee63751ed4f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:58.703759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.703701 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.703759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.703723 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.703759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.703733 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5023934-6d80-4502-8b04-ee63751ed4f7-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:58.703759 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:58.703742 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b292s\" (UniqueName: \"kubernetes.io/projected/b5023934-6d80-4502-8b04-ee63751ed4f7-kube-api-access-b292s\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:59:59.051394 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.051272 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:59.162656 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.162624 5973 generic.go:356] "Generic (PLEG): container finished" podID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerID="8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77" exitCode=0 Nov 03 21:59:59.162843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.162684 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" Nov 03 21:59:59.162843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.162714 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerDied","Data":"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77"} Nov 03 21:59:59.162843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.162765 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf" event={"ID":"b5023934-6d80-4502-8b04-ee63751ed4f7","Type":"ContainerDied","Data":"a151470e35c6adc0235f06965f5cb0fce6779c8f0a8dd92e865d6833b9a24326"} Nov 03 21:59:59.162843 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.162797 5973 scope.go:122] "RemoveContainer" containerID="8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77" Nov 03 21:59:59.176331 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.175682 5973 scope.go:122] "RemoveContainer" containerID="c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83" Nov 03 21:59:59.192055 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.192030 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:59:59.193753 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.193730 5973 scope.go:122] "RemoveContainer" containerID="8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77" Nov 03 21:59:59.194177 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:59.194154 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77\": container with ID starting with 8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77 not found: ID does not exist" containerID="8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77" Nov 03 21:59:59.194288 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.194186 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77"} err="failed to get container status \"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77\": rpc error: code = NotFound desc = could not find container \"8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77\": container with ID starting with 8c472d4a50db26fa488333c54261d3801234f505472f17fba7800312311baf77 not found: ID does not exist" Nov 03 21:59:59.194288 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.194210 5973 scope.go:122] "RemoveContainer" containerID="c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83" Nov 03 21:59:59.194919 ip-10-0-131-89 kubenswrapper[5973]: E1103 21:59:59.194897 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83\": container with ID starting with c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83 not found: ID does not exist" containerID="c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83" Nov 03 21:59:59.194987 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.194929 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83"} err="failed to get container status \"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83\": rpc error: code = NotFound desc = could not find container \"c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83\": container with ID starting with c4b2886bb75faa04a34d854d49da8be499855c11f80599e376507830d83fed83 not found: ID does not exist" Nov 03 21:59:59.195855 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.195835 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f4fb1b-openshift-default-78b8dffc77-gzvxf"] Nov 03 21:59:59.767717 ip-10-0-131-89 kubenswrapper[5973]: I1103 21:59:59.767679 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:00.156574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.156528 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:00.157120 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157089 5973 cpu_manager.go:414] "containerMap: removing container" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.157120 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157118 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157137 5973 cpu_manager.go:414] "containerMap: removing container" podUID="fba4ca7f-bbd2-4280-8598-ef50d031b42e" containerName="revoke-keys" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157145 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba4ca7f-bbd2-4280-8598-ef50d031b42e" containerName="revoke-keys" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157158 5973 cpu_manager.go:414] "containerMap: removing container" podUID="58d7f37b-404a-4dbb-8971-43201013b7ad" containerName="revoke-keys" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157165 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d7f37b-404a-4dbb-8971-43201013b7ad" containerName="revoke-keys" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157191 5973 cpu_manager.go:414] "containerMap: removing container" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157197 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157217 5973 cpu_manager.go:414] "containerMap: removing container" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157224 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157237 5973 cpu_manager.go:414] "containerMap: removing container" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.157272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157244 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157363 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157375 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba4ca7f-bbd2-4280-8598-ef50d031b42e" containerName="revoke-keys" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157384 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d7f37b-404a-4dbb-8971-43201013b7ad" containerName="revoke-keys" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157397 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37b4367-bffb-407c-896e-0c2b463fafd4" containerName="istio-proxy" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157409 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.157774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.157418 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="7290c8e9-990d-4032-96e3-501b933cdc9b" containerName="istio-proxy" Nov 03 22:00:00.162777 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.162754 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:00.162896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.162877 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.163228 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.163213 5973 cpu_manager.go:414] "containerMap: removing container" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.163319 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.163231 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.163401 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.163389 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" containerName="istio-proxy" Nov 03 22:00:00.166562 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.166539 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.168581 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.168559 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:00:00.176779 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.176756 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:00.181818 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.181785 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:00.214979 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.214945 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:00:00.215146 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.215001 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:00:00.215435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.215407 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" podUID="862511e9-381b-4710-8587-f0a9abbe9caf" containerName="main" containerID="cri-o://3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63" gracePeriod=45 Nov 03 22:00:00.322592 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.322559 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.322780 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.322629 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.322780 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.322715 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62gn\" (UniqueName: \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.322943 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.322924 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85lh\" (UniqueName: \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.346831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.346799 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5023934-6d80-4502-8b04-ee63751ed4f7" path="/var/lib/kubelet/pods/b5023934-6d80-4502-8b04-ee63751ed4f7/volumes" Nov 03 22:00:00.424607 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.424522 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85lh\" (UniqueName: \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.424771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.424631 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.424771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.424694 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.424771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.424718 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62gn\" (UniqueName: \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.427516 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.427495 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:00:00.435743 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.435708 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.435847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.435777 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.437268 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.437251 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:00:00.447412 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.447381 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62gn\" (UniqueName: \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\") pod \"maas-api-key-cleanup-29895720-5mb44\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.447412 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.447406 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85lh\" (UniqueName: \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\") pod \"maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.493622 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.493597 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:00:00.496028 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.496003 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 22:00:00.497749 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.497729 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:00.500665 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.500593 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:00.500726 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.500700 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:00.628171 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628135 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628200 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\" (UniqueName: \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628274 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628469 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628337 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628469 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628374 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628469 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628398 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:00.628469 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628413 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\" (UniqueName: \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\") pod \"862511e9-381b-4710-8587-f0a9abbe9caf\" (UID: \"862511e9-381b-4710-8587-f0a9abbe9caf\") " Nov 03 22:00:00.628699 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628676 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:00.628944 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628924 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:00.629011 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628945 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.629011 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.628963 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.630981 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.630955 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:00.631804 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.631774 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "kube-api-access-7jsnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:00.632664 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.632646 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs" pod "862511e9-381b-4710-8587-f0a9abbe9caf" (UID: "862511e9-381b-4710-8587-f0a9abbe9caf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:00.659310 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.658866 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:00.730902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.730802 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862511e9-381b-4710-8587-f0a9abbe9caf-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.730902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.730842 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.730902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.730855 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862511e9-381b-4710-8587-f0a9abbe9caf-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.730902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.730869 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsnh\" (UniqueName: \"kubernetes.io/projected/862511e9-381b-4710-8587-f0a9abbe9caf-kube-api-access-7jsnh\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:00.800967 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.800936 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:00.801322 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:00:00.801158 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a0eeaa_c402_4ab8_ba8d_8f8db8784e64.slice/crio-23e54fe12f3f740e58f4f6cba34a011e0bbe91464f24a03ef9c566c4f045b1b7 WatchSource:0}: Error finding container 23e54fe12f3f740e58f4f6cba34a011e0bbe91464f24a03ef9c566c4f045b1b7: Status 404 returned error can't find the container with id 23e54fe12f3f740e58f4f6cba34a011e0bbe91464f24a03ef9c566c4f045b1b7 Nov 03 22:00:00.823400 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.818251 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:00.842246 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.841946 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:00.916948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.916642 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 22:00:00.930357 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.930319 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:00:00.988344 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.988256 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc"] Nov 03 22:00:00.988979 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.988958 5973 cpu_manager.go:414] "containerMap: removing container" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989094 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.988981 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989094 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989002 5973 cpu_manager.go:414] "containerMap: removing container" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989094 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989012 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989094 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989030 5973 cpu_manager.go:414] "containerMap: removing container" podUID="862511e9-381b-4710-8587-f0a9abbe9caf" containerName="main" Nov 03 22:00:00.989094 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989037 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862511e9-381b-4710-8587-f0a9abbe9caf" containerName="main" Nov 03 22:00:00.989407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989141 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989154 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="324bdd22-26a5-4724-9d87-4c369423398d" containerName="istio-proxy" Nov 03 22:00:00.989407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.989165 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="862511e9-381b-4710-8587-f0a9abbe9caf" containerName="main" Nov 03 22:00:00.995585 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:00.994817 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.020333 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.020287 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc"] Nov 03 22:00:01.024428 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.022832 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.033759 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.033821 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.033941 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.033975 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.034061 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.034093 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.034142 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.034179 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.034209 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\") pod \"324bdd22-26a5-4724-9d87-4c369423398d\" (UID: \"324bdd22-26a5-4724-9d87-4c369423398d\") " Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.035205 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.035388 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.035504 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:01.052718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.035705 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:01.053417 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.053013 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:01.062598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.062564 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:01.062598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.062577 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:01.062765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.062564 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:01.062765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.062577 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6" pod "324bdd22-26a5-4724-9d87-4c369423398d" (UID: "324bdd22-26a5-4724-9d87-4c369423398d"). InnerVolumeSpecName "kube-api-access-49zq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:01.089449 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.089419 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:00:01.135167 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135123 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.135356 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135245 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.135356 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135306 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ccw\" (UniqueName: \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.135435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135388 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zq6\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-kube-api-access-49zq6\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135401 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135410 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135448 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135491 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135500 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324bdd22-26a5-4724-9d87-4c369423398d-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135509 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324bdd22-26a5-4724-9d87-4c369423398d-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135517 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324bdd22-26a5-4724-9d87-4c369423398d-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.135574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.135526 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324bdd22-26a5-4724-9d87-4c369423398d-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:01.198844 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.198810 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerStarted","Data":"93c87ce4a759d09b648770230803edaca8726a668447b0239f1d7e5d9468cecc"} Nov 03 22:00:01.199047 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.198897 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerStarted","Data":"23e54fe12f3f740e58f4f6cba34a011e0bbe91464f24a03ef9c566c4f045b1b7"} Nov 03 22:00:01.200703 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.200678 5973 generic.go:356] "Generic (PLEG): container finished" podID="324bdd22-26a5-4724-9d87-4c369423398d" containerID="c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0" exitCode=0 Nov 03 22:00:01.200823 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.200757 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerDied","Data":"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0"} Nov 03 22:00:01.200823 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.200780 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" Nov 03 22:00:01.200823 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.200803 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb" event={"ID":"324bdd22-26a5-4724-9d87-4c369423398d","Type":"ContainerDied","Data":"d5e373253404c8ac32e1d43f0f33eaf7a3d3a48bf06750602582bb829029a424"} Nov 03 22:00:01.200958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.200834 5973 scope.go:122] "RemoveContainer" containerID="c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0" Nov 03 22:00:01.206019 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.205988 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" event={"ID":"6e508779-36bd-4777-aa76-beffba2b7c0d","Type":"ContainerStarted","Data":"dc832fbffe9aae0769402b417403e4de291871bed2a44530cfa98d776fb1b348"} Nov 03 22:00:01.207485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.207439 5973 generic.go:356] "Generic (PLEG): container finished" podID="862511e9-381b-4710-8587-f0a9abbe9caf" containerID="3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63" exitCode=0 Nov 03 22:00:01.207569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.207491 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" event={"ID":"862511e9-381b-4710-8587-f0a9abbe9caf","Type":"ContainerDied","Data":"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63"} Nov 03 22:00:01.207569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.207514 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" Nov 03 22:00:01.207569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.207529 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc" event={"ID":"862511e9-381b-4710-8587-f0a9abbe9caf","Type":"ContainerDied","Data":"04b588029b33aff2223822d7b6835cc1ccf5ad7391a98e8065e54b1ed520ae35"} Nov 03 22:00:01.213406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.213383 5973 scope.go:122] "RemoveContainer" containerID="5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13" Nov 03 22:00:01.219138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.219092 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" podStartSLOduration=1.219076392 podStartE2EDuration="1.219076392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.217617313 +0000 UTC m=+840.992467161" watchObservedRunningTime="2026-11-03 22:00:01.219076392 +0000 UTC m=+840.993926238" Nov 03 22:00:01.225047 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.224920 5973 scope.go:122] "RemoveContainer" containerID="c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0" Nov 03 22:00:01.225296 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:01.225271 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0\": container with ID starting with c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0 not found: ID does not exist" containerID="c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0" Nov 03 22:00:01.225352 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.225308 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0"} err="failed to get container status \"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0\": rpc error: code = NotFound desc = could not find container \"c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0\": container with ID starting with c5609449df173bd8de1d69631a50904eacd2a459cab1c77708b089f071b125d0 not found: ID does not exist" Nov 03 22:00:01.225352 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.225326 5973 scope.go:122] "RemoveContainer" containerID="5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13" Nov 03 22:00:01.225666 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:01.225633 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13\": container with ID starting with 5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13 not found: ID does not exist" containerID="5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13" Nov 03 22:00:01.225752 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.225667 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13"} err="failed to get container status \"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13\": rpc error: code = NotFound desc = could not find container \"5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13\": container with ID starting with 5c211667c8bbccaf2d81cb349cbb9363c4afa2fa2d80cf6674161ffbdaf61c13 not found: ID does not exist" Nov 03 22:00:01.225752 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.225683 5973 scope.go:122] "RemoveContainer" containerID="3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63" Nov 03 22:00:01.237331 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.237300 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ccw\" (UniqueName: \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.237499 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.237438 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.237588 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.237572 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.238312 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.238286 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.240527 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.240476 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.240610 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.240529 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 22:00:01.248417 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.248393 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8af3f3-openshift-default-56b456db75-vxtdb"] Nov 03 22:00:01.249857 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.249837 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ccw\" (UniqueName: \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\") pod \"maas-api-revoke-sub-9c26b79c-r6xrc\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.258215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.258190 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 22:00:01.264034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.264010 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc16de/premium-w4-fc16de-kserve-5479d48f79-4v9wc"] Nov 03 22:00:01.272225 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.272207 5973 scope.go:122] "RemoveContainer" containerID="3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63" Nov 03 22:00:01.272554 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:01.272534 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63\": container with ID starting with 3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63 not found: ID does not exist" containerID="3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63" Nov 03 22:00:01.272612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.272565 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63"} err="failed to get container status \"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63\": rpc error: code = NotFound desc = could not find container \"3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63\": container with ID starting with 3d8ce98cba1e90dcf2bc1c3592ec3e0b4cce7ff0d4696b4c379548bb593b2d63 not found: ID does not exist" Nov 03 22:00:01.355286 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.355255 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:01.675583 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:00:01.675549 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390589f8_71d7_451e_8728_13447c6be485.slice/crio-3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7 WatchSource:0}: Error finding container 3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7: Status 404 returned error can't find the container with id 3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7 Nov 03 22:00:01.676259 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.676233 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc"] Nov 03 22:00:01.963213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:01.963186 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 22:00:02.046369 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046329 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046381 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046418 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046533 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046568 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046820 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046701 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046820 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046765 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046855 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.046928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046852 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:02.046928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046901 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\") pod \"e63f36a1-ff07-4731-ad73-c6f9502e1460\" (UID: \"e63f36a1-ff07-4731-ad73-c6f9502e1460\") " Nov 03 22:00:02.047057 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.046963 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:02.047057 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047034 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:02.047327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047074 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:02.047327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047250 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:02.047449 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047427 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.047538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047472 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.047538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047487 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.047538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047500 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63f36a1-ff07-4731-ad73-c6f9502e1460-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.047538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.047514 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.048880 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.048853 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:02.048880 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.048870 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:02.049026 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.048958 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "kube-api-access-sf5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:02.049121 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.049094 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token" pod "e63f36a1-ff07-4731-ad73-c6f9502e1460" (UID: "e63f36a1-ff07-4731-ad73-c6f9502e1460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:02.149160 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.149128 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.149160 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.149155 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.149160 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.149166 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63f36a1-ff07-4731-ad73-c6f9502e1460-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.149390 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.149175 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf5n2\" (UniqueName: \"kubernetes.io/projected/e63f36a1-ff07-4731-ad73-c6f9502e1460-kube-api-access-sf5n2\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:02.217533 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.217427 5973 generic.go:356] "Generic (PLEG): container finished" podID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerID="43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379" exitCode=0 Nov 03 22:00:02.217533 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.217490 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" Nov 03 22:00:02.217533 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.217490 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerDied","Data":"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379"} Nov 03 22:00:02.217804 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.217555 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr" event={"ID":"e63f36a1-ff07-4731-ad73-c6f9502e1460","Type":"ContainerDied","Data":"599ca8faf34436f058e47a3aa4ba13ef007bea26877e8cafde18b1ad0bd5aaad"} Nov 03 22:00:02.217804 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.217586 5973 scope.go:122] "RemoveContainer" containerID="43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379" Nov 03 22:00:02.224357 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.219806 5973 generic.go:356] "Generic (PLEG): container finished" podID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerID="db23bc3975de30a053c6bbc9684972ef1ccc1027dbad218a5f5ec9c9f98d87ae" exitCode=60 Nov 03 22:00:02.224357 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.219922 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" event={"ID":"6e508779-36bd-4777-aa76-beffba2b7c0d","Type":"ContainerDied","Data":"db23bc3975de30a053c6bbc9684972ef1ccc1027dbad218a5f5ec9c9f98d87ae"} Nov 03 22:00:02.224357 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.220277 5973 scope.go:122] "RemoveContainer" containerID="db23bc3975de30a053c6bbc9684972ef1ccc1027dbad218a5f5ec9c9f98d87ae" Nov 03 22:00:02.230647 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.230620 5973 generic.go:356] "Generic (PLEG): container finished" podID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerID="93c87ce4a759d09b648770230803edaca8726a668447b0239f1d7e5d9468cecc" exitCode=60 Nov 03 22:00:02.230766 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.230695 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerDied","Data":"93c87ce4a759d09b648770230803edaca8726a668447b0239f1d7e5d9468cecc"} Nov 03 22:00:02.230921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.230903 5973 scope.go:122] "RemoveContainer" containerID="93c87ce4a759d09b648770230803edaca8726a668447b0239f1d7e5d9468cecc" Nov 03 22:00:02.232584 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.232565 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" event={"ID":"390589f8-71d7-451e-8728-13447c6be485","Type":"ContainerStarted","Data":"884e0fdaff98a8d36976c655914e22670c9f74c6fbc1bb4a7d3806ece77c846f"} Nov 03 22:00:02.232671 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.232606 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" event={"ID":"390589f8-71d7-451e-8728-13447c6be485","Type":"ContainerStarted","Data":"3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7"} Nov 03 22:00:02.249558 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.249535 5973 scope.go:122] "RemoveContainer" containerID="c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb" Nov 03 22:00:02.265063 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.265019 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" podStartSLOduration=2.265002766 podStartE2EDuration="2.265002766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.261052684 +0000 UTC m=+842.035902533" watchObservedRunningTime="2026-11-03 22:00:02.265002766 +0000 UTC m=+842.039852614" Nov 03 22:00:02.269711 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.269696 5973 scope.go:122] "RemoveContainer" containerID="43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379" Nov 03 22:00:02.270028 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:02.270011 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379\": container with ID starting with 43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379 not found: ID does not exist" containerID="43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379" Nov 03 22:00:02.270064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.270048 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379"} err="failed to get container status \"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379\": rpc error: code = NotFound desc = could not find container \"43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379\": container with ID starting with 43a280f3b416749c434fc514320642203436b9bb6af3f990435239f997504379 not found: ID does not exist" Nov 03 22:00:02.270099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.270066 5973 scope.go:122] "RemoveContainer" containerID="c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb" Nov 03 22:00:02.270316 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:02.270302 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb\": container with ID starting with c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb not found: ID does not exist" containerID="c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb" Nov 03 22:00:02.270352 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.270321 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb"} err="failed to get container status \"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb\": rpc error: code = NotFound desc = could not find container \"c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb\": container with ID starting with c80f774cf0d679abd1e9ade108f41545bd0e2283c568def9e470519070ff51bb not found: ID does not exist" Nov 03 22:00:02.298360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.298335 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 22:00:02.306118 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.306093 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-02e8ab-openshift-default-6f4974886c-d5kcr"] Nov 03 22:00:02.346282 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.346254 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324bdd22-26a5-4724-9d87-4c369423398d" path="/var/lib/kubelet/pods/324bdd22-26a5-4724-9d87-4c369423398d/volumes" Nov 03 22:00:02.346699 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.346686 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862511e9-381b-4710-8587-f0a9abbe9caf" path="/var/lib/kubelet/pods/862511e9-381b-4710-8587-f0a9abbe9caf/volumes" Nov 03 22:00:02.347065 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:02.347053 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" path="/var/lib/kubelet/pods/e63f36a1-ff07-4731-ad73-c6f9502e1460/volumes" Nov 03 22:00:03.242368 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.242333 5973 generic.go:356] "Generic (PLEG): container finished" podID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerID="d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647" exitCode=60 Nov 03 22:00:03.242829 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.242423 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerDied","Data":"d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647"} Nov 03 22:00:03.242829 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.242507 5973 scope.go:122] "RemoveContainer" containerID="93c87ce4a759d09b648770230803edaca8726a668447b0239f1d7e5d9468cecc" Nov 03 22:00:03.242829 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.242793 5973 scope.go:122] "RemoveContainer" containerID="d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647" Nov 03 22:00:03.243111 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:03.243088 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-5mb44_odh-ai-gateway-infra(55a0eeaa-c402-4ab8-ba8d-8f8db8784e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" Nov 03 22:00:03.244299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.244277 5973 generic.go:356] "Generic (PLEG): container finished" podID="390589f8-71d7-451e-8728-13447c6be485" containerID="884e0fdaff98a8d36976c655914e22670c9f74c6fbc1bb4a7d3806ece77c846f" exitCode=0 Nov 03 22:00:03.244404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.244362 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" event={"ID":"390589f8-71d7-451e-8728-13447c6be485","Type":"ContainerDied","Data":"884e0fdaff98a8d36976c655914e22670c9f74c6fbc1bb4a7d3806ece77c846f"} Nov 03 22:00:03.247337 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.247314 5973 generic.go:356] "Generic (PLEG): container finished" podID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerID="500193f2dd3e1d4edf72d3021df891b75341e4fc57c3216f000084cae178e465" exitCode=60 Nov 03 22:00:03.247424 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.247398 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" event={"ID":"6e508779-36bd-4777-aa76-beffba2b7c0d","Type":"ContainerDied","Data":"500193f2dd3e1d4edf72d3021df891b75341e4fc57c3216f000084cae178e465"} Nov 03 22:00:03.247618 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.247604 5973 scope.go:122] "RemoveContainer" containerID="500193f2dd3e1d4edf72d3021df891b75341e4fc57c3216f000084cae178e465" Nov 03 22:00:03.247836 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:03.247817 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk_odh-ai-gateway-infra(6e508779-36bd-4777-aa76-beffba2b7c0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" Nov 03 22:00:03.255217 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.255201 5973 scope.go:122] "RemoveContainer" containerID="db23bc3975de30a053c6bbc9684972ef1ccc1027dbad218a5f5ec9c9f98d87ae" Nov 03 22:00:03.818514 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:03.818449 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.102082 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.101977 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.256700 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.256670 5973 scope.go:122] "RemoveContainer" containerID="500193f2dd3e1d4edf72d3021df891b75341e4fc57c3216f000084cae178e465" Nov 03 22:00:04.257126 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:04.256966 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk_odh-ai-gateway-infra(6e508779-36bd-4777-aa76-beffba2b7c0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" Nov 03 22:00:04.257986 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.257960 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.259083 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.259058 5973 scope.go:122] "RemoveContainer" containerID="d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647" Nov 03 22:00:04.259526 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:04.259287 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-5mb44_odh-ai-gateway-infra(55a0eeaa-c402-4ab8-ba8d-8f8db8784e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" Nov 03 22:00:04.403522 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.403490 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:04.427406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.427372 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.574207 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.574177 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\") pod \"390589f8-71d7-451e-8728-13447c6be485\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " Nov 03 22:00:04.574445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.574310 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\") pod \"390589f8-71d7-451e-8728-13447c6be485\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " Nov 03 22:00:04.574445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.574354 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\" (UniqueName: \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\") pod \"390589f8-71d7-451e-8728-13447c6be485\" (UID: \"390589f8-71d7-451e-8728-13447c6be485\") " Nov 03 22:00:04.577661 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.577631 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca" pod "390589f8-71d7-451e-8728-13447c6be485" (UID: "390589f8-71d7-451e-8728-13447c6be485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:04.579602 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.579577 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw" pod "390589f8-71d7-451e-8728-13447c6be485" (UID: "390589f8-71d7-451e-8728-13447c6be485"). InnerVolumeSpecName "kube-api-access-f4ccw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:04.585964 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.585937 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca" pod "390589f8-71d7-451e-8728-13447c6be485" (UID: "390589f8-71d7-451e-8728-13447c6be485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:04.676591 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.676545 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390589f8-71d7-451e-8728-13447c6be485-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.676591 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.676583 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390589f8-71d7-451e-8728-13447c6be485-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.676591 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.676597 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ccw\" (UniqueName: \"kubernetes.io/projected/390589f8-71d7-451e-8728-13447c6be485-kube-api-access-f4ccw\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:04.683518 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.683451 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:04.820793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.820760 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm"] Nov 03 22:00:04.824600 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.824570 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc16de-6f9sm"] Nov 03 22:00:04.856741 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:04.856696 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:05.024171 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.024087 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:05.215707 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.215667 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:05.271224 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.271193 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" Nov 03 22:00:05.271224 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.271217 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc" event={"ID":"390589f8-71d7-451e-8728-13447c6be485","Type":"ContainerDied","Data":"3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7"} Nov 03 22:00:05.271778 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.271264 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc237e53c5bd97a2299d1e5fb49d5a6560f284f95bd19508fd7aace4a50b7a7" Nov 03 22:00:05.370420 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.370372 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:05.520097 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:05.520065 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.020171 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.020139 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.191997 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.191960 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.345388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.345285 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab378c7-6261-4468-9c6d-cf62cafbc4fb" path="/var/lib/kubelet/pods/0ab378c7-6261-4468-9c6d-cf62cafbc4fb/volumes" Nov 03 22:00:06.566289 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.566252 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:06.823668 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.823637 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:06.824066 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824053 5973 cpu_manager.go:414] "containerMap: removing container" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824067 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824078 5973 cpu_manager.go:414] "containerMap: removing container" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824083 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824098 5973 cpu_manager.go:414] "containerMap: removing container" podUID="390589f8-71d7-451e-8728-13447c6be485" containerName="revoke-keys" Nov 03 22:00:06.824117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824103 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390589f8-71d7-451e-8728-13447c6be485" containerName="revoke-keys" Nov 03 22:00:06.824274 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824166 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824274 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824172 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63f36a1-ff07-4731-ad73-c6f9502e1460" containerName="istio-proxy" Nov 03 22:00:06.824274 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.824180 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="390589f8-71d7-451e-8728-13447c6be485" containerName="revoke-keys" Nov 03 22:00:06.827773 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.827754 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.896776 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.896738 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:06.900709 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.900678 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5hb\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.900885 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.900716 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.900885 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.900816 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901025 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.900931 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901025 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.900983 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901134 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.901027 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901134 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.901049 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901233 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.901152 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:06.901233 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:06.901211 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.001719 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001682 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.001873 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001728 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.001873 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001768 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.001873 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001815 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.001873 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001867 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5hb\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001899 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001947 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.001994 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002032 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002245 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002145 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002327 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002514 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002408 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002514 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002421 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.002896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.002875 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.004446 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.004422 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.004717 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.004698 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.017272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.017237 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5hb\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.020107 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.020081 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\") pod \"e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.141036 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.140947 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:07.426586 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.426559 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:07.427113 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:00:07.427088 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5180836_2929_4ef0_b4c5_36d20d1df3b5.slice/crio-544de295431de65f281496f12b8eea5e0541055808af1beb180c585bea29a459 WatchSource:0}: Error finding container 544de295431de65f281496f12b8eea5e0541055808af1beb180c585bea29a459: Status 404 returned error can't find the container with id 544de295431de65f281496f12b8eea5e0541055808af1beb180c585bea29a459 Nov 03 22:00:07.429399 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.429364 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:07.429526 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.429445 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:07.429526 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:07.429508 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:08.308737 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:08.308697 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" event={"ID":"f5180836-2929-4ef0-b4c5-36d20d1df3b5","Type":"ContainerStarted","Data":"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf"} Nov 03 22:00:08.308906 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:08.308752 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" event={"ID":"f5180836-2929-4ef0-b4c5-36d20d1df3b5","Type":"ContainerStarted","Data":"544de295431de65f281496f12b8eea5e0541055808af1beb180c585bea29a459"} Nov 03 22:00:08.338115 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:08.338050 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" podStartSLOduration=2.33803611 podStartE2EDuration="2.33803611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:08.334653264 +0000 UTC m=+848.109503166" watchObservedRunningTime="2026-11-03 22:00:08.33803611 +0000 UTC m=+848.112885956" Nov 03 22:00:09.141717 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:09.141674 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:09.147032 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:09.147009 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:09.315183 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:09.315151 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:09.316093 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:09.316069 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:09.598202 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:09.598111 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:10.403420 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.403363 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerName="main" containerID="cri-o://d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9" gracePeriod=45 Nov 03 22:00:10.553723 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.553687 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused" Nov 03 22:00:10.632142 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.632110 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:00:10.658254 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.658195 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 22:00:10.735707 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735673 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\" (UniqueName: \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.735886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735718 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.735886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735764 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.735886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735844 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.735886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735877 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.736112 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.735912 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\") pod \"ab0ed1f9-7468-479c-87ef-b60024d86db4\" (UID: \"ab0ed1f9-7468-479c-87ef-b60024d86db4\") " Nov 03 22:00:10.736218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.736196 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:10.736310 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.736295 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:10.736794 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.736770 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:10.736884 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.736835 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:10.738502 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.738474 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:10.738708 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.738684 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:10.738932 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.738903 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6" pod "ab0ed1f9-7468-479c-87ef-b60024d86db4" (UID: "ab0ed1f9-7468-479c-87ef-b60024d86db4"). InnerVolumeSpecName "kube-api-access-72bw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:10.837555 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.837520 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:10.837555 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.837547 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:10.837555 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.837558 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bw6\" (UniqueName: \"kubernetes.io/projected/ab0ed1f9-7468-479c-87ef-b60024d86db4-kube-api-access-72bw6\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:10.837793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.837570 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0ed1f9-7468-479c-87ef-b60024d86db4-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:10.837793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:10.837580 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0ed1f9-7468-479c-87ef-b60024d86db4-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:11.240810 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.240773 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:11.328214 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.328181 5973 generic.go:356] "Generic (PLEG): container finished" podID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerID="d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9" exitCode=0 Nov 03 22:00:11.328388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.328223 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" Nov 03 22:00:11.328388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.328267 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" event={"ID":"ab0ed1f9-7468-479c-87ef-b60024d86db4","Type":"ContainerDied","Data":"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9"} Nov 03 22:00:11.328388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.328314 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz" event={"ID":"ab0ed1f9-7468-479c-87ef-b60024d86db4","Type":"ContainerDied","Data":"3bade80c13a23807ed9c71f7592c32d7ab69b2df4ae77b79eef921ab4de46e57"} Nov 03 22:00:11.328388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.328337 5973 scope.go:122] "RemoveContainer" containerID="d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9" Nov 03 22:00:11.348673 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.348641 5973 scope.go:122] "RemoveContainer" containerID="d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9" Nov 03 22:00:11.349204 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:11.349178 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9\": container with ID starting with d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9 not found: ID does not exist" containerID="d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9" Nov 03 22:00:11.349303 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.349210 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9"} err="failed to get container status \"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9\": rpc error: code = NotFound desc = could not find container \"d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9\": container with ID starting with d5fe2af25ec67372d2f0e2d70f5064795a23e0d77a5b22d89c06c0df7698d8b9 not found: ID does not exist" Nov 03 22:00:11.355496 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.355473 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 22:00:11.361096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.361072 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-53a2cb/simulator-w2-53a2cb-kserve-76c84bfd8c-lr9dz"] Nov 03 22:00:11.748194 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.748160 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:11.916694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:11.916661 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:12.028035 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.027941 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 22:00:12.028577 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.028212 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" containerID="cri-o://869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da" gracePeriod=30 Nov 03 22:00:12.032079 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.032051 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:12.136552 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.136514 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:12.286576 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.286490 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:12.336681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.336644 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" podUID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" containerName="istio-proxy" containerID="cri-o://85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf" gracePeriod=30 Nov 03 22:00:12.345519 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.345494 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" path="/var/lib/kubelet/pods/ab0ed1f9-7468-479c-87ef-b60024d86db4/volumes" Nov 03 22:00:12.712753 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:12.712720 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:14.339307 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:14.339278 5973 scope.go:122] "RemoveContainer" containerID="d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647" Nov 03 22:00:14.401559 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:14.401521 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:00:15.099620 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.099538 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:15.172290 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.172250 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:15.238627 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.238601 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:15.281349 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.281318 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\") pod \"6e508779-36bd-4777-aa76-beffba2b7c0d\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " Nov 03 22:00:15.281598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.281479 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\" (UniqueName: \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\") pod \"6e508779-36bd-4777-aa76-beffba2b7c0d\" (UID: \"6e508779-36bd-4777-aa76-beffba2b7c0d\") " Nov 03 22:00:15.282267 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.281792 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca" pod "6e508779-36bd-4777-aa76-beffba2b7c0d" (UID: "6e508779-36bd-4777-aa76-beffba2b7c0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:15.283519 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.282790 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e508779-36bd-4777-aa76-beffba2b7c0d-maas-api-service-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:15.285121 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.284376 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh" pod "6e508779-36bd-4777-aa76-beffba2b7c0d" (UID: "6e508779-36bd-4777-aa76-beffba2b7c0d"). InnerVolumeSpecName "kube-api-access-z85lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:15.291927 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.291884 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh"] Nov 03 22:00:15.294215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.294189 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-53a2cb-v92fh"] Nov 03 22:00:15.361030 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.360921 5973 generic.go:356] "Generic (PLEG): container finished" podID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerID="51c8437a82acbd9cb5308e88ae0ba1508ac1a145f6c80585cd33453dd32d529f" exitCode=60 Nov 03 22:00:15.361560 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.361131 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerDied","Data":"51c8437a82acbd9cb5308e88ae0ba1508ac1a145f6c80585cd33453dd32d529f"} Nov 03 22:00:15.361560 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.361190 5973 scope.go:122] "RemoveContainer" containerID="d72190101a941dd5e7642017a8b79fc9c00c270a2a0ea1398d764c046ffdc647" Nov 03 22:00:15.361691 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.361674 5973 scope.go:122] "RemoveContainer" containerID="51c8437a82acbd9cb5308e88ae0ba1508ac1a145f6c80585cd33453dd32d529f" Nov 03 22:00:15.361999 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:15.361964 5973 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-5mb44_odh-ai-gateway-infra(55a0eeaa-c402-4ab8-ba8d-8f8db8784e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" Nov 03 22:00:15.366018 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.365985 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" event={"ID":"6e508779-36bd-4777-aa76-beffba2b7c0d","Type":"ContainerDied","Data":"dc832fbffe9aae0769402b417403e4de291871bed2a44530cfa98d776fb1b348"} Nov 03 22:00:15.366165 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.366149 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk" Nov 03 22:00:15.385615 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.385358 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85lh\" (UniqueName: \"kubernetes.io/projected/6e508779-36bd-4777-aa76-beffba2b7c0d-kube-api-access-z85lh\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:15.388388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.388363 5973 scope.go:122] "RemoveContainer" containerID="500193f2dd3e1d4edf72d3021df891b75341e4fc57c3216f000084cae178e465" Nov 03 22:00:15.410199 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.410173 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:15.416158 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.416136 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-53a2cb-29895720-vdkwk"] Nov 03 22:00:15.580015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.579979 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:15.736641 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.736605 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:15.891429 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:15.891396 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:16.345154 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.345124 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d7f37b-404a-4dbb-8971-43201013b7ad" path="/var/lib/kubelet/pods/58d7f37b-404a-4dbb-8971-43201013b7ad/volumes" Nov 03 22:00:16.345541 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.345527 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" path="/var/lib/kubelet/pods/6e508779-36bd-4777-aa76-beffba2b7c0d/volumes" Nov 03 22:00:16.393045 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.393017 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:16.541283 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.541258 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:16.596316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.596226 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\") pod \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " Nov 03 22:00:16.596316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.596272 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\" (UniqueName: \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\") pod \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\" (UID: \"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64\") " Nov 03 22:00:16.596665 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.596640 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca" pod "55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" (UID: "55a0eeaa-c402-4ab8-ba8d-8f8db8784e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:16.598750 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.598719 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn" pod "55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" (UID: "55a0eeaa-c402-4ab8-ba8d-8f8db8784e64"). InnerVolumeSpecName "kube-api-access-t62gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:16.697339 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.697306 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-maas-api-service-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:16.697339 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.697332 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t62gn\" (UniqueName: \"kubernetes.io/projected/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64-kube-api-access-t62gn\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:16.924819 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:16.924787 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:00:17.289776 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.289752 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 22:00:17.383133 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.383105 5973 generic.go:356] "Generic (PLEG): container finished" podID="c50978f4-39bd-4045-8c75-2ba431f257de" containerID="869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da" exitCode=0 Nov 03 22:00:17.383266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.383155 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" Nov 03 22:00:17.383266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.383192 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerDied","Data":"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da"} Nov 03 22:00:17.383266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.383248 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc" event={"ID":"c50978f4-39bd-4045-8c75-2ba431f257de","Type":"ContainerDied","Data":"f21caced290e170143fbc6a503e531d531b7127037573441853e6b0769abdcd4"} Nov 03 22:00:17.383440 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.383278 5973 scope.go:122] "RemoveContainer" containerID="869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da" Nov 03 22:00:17.384502 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.384484 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" Nov 03 22:00:17.384502 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.384496 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44" event={"ID":"55a0eeaa-c402-4ab8-ba8d-8f8db8784e64","Type":"ContainerDied","Data":"23e54fe12f3f740e58f4f6cba34a011e0bbe91464f24a03ef9c566c4f045b1b7"} Nov 03 22:00:17.406495 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406440 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406522 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406559 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406584 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406613 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406665 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406690 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406720 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406763 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.406891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.406800 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\") pod \"c50978f4-39bd-4045-8c75-2ba431f257de\" (UID: \"c50978f4-39bd-4045-8c75-2ba431f257de\") " Nov 03 22:00:17.407474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407112 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.407474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407183 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.407608 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407501 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.407608 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407551 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.407608 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407566 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.407766 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.407658 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:17.408080 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.408054 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.409112 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.409079 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.410043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.410022 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:17.410305 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.410287 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "kube-api-access-rlprr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:17.410404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.410391 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token" pod "c50978f4-39bd-4045-8c75-2ba431f257de" (UID: "c50978f4-39bd-4045-8c75-2ba431f257de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:17.426166 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.426141 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:17.430736 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.430717 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-5mb44"] Nov 03 22:00:17.449710 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.449690 5973 scope.go:122] "RemoveContainer" containerID="e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62" Nov 03 22:00:17.459266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.459246 5973 scope.go:122] "RemoveContainer" containerID="869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da" Nov 03 22:00:17.459546 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:17.459527 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da\": container with ID starting with 869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da not found: ID does not exist" containerID="869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da" Nov 03 22:00:17.459616 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.459585 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da"} err="failed to get container status \"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da\": rpc error: code = NotFound desc = could not find container \"869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da\": container with ID starting with 869741066e138ac1cc0352950445490f6cf7f88ad1b5f7361c303fa5aa6574da not found: ID does not exist" Nov 03 22:00:17.459616 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.459607 5973 scope.go:122] "RemoveContainer" containerID="e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62" Nov 03 22:00:17.459884 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:17.459857 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62\": container with ID starting with e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62 not found: ID does not exist" containerID="e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62" Nov 03 22:00:17.459931 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.459889 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62"} err="failed to get container status \"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62\": rpc error: code = NotFound desc = could not find container \"e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62\": container with ID starting with e0c621ba6b7aca28db686ea6b588c9c6abf0cc64bde58406d43b0cfc26e60f62 not found: ID does not exist" Nov 03 22:00:17.459931 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.459905 5973 scope.go:122] "RemoveContainer" containerID="51c8437a82acbd9cb5308e88ae0ba1508ac1a145f6c80585cd33453dd32d529f" Nov 03 22:00:17.508143 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508114 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50978f4-39bd-4045-8c75-2ba431f257de-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.508143 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508140 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50978f4-39bd-4045-8c75-2ba431f257de-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.508143 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508151 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlprr\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-kube-api-access-rlprr\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.508366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508160 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.508366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508168 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50978f4-39bd-4045-8c75-2ba431f257de-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.508366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.508175 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50978f4-39bd-4045-8c75-2ba431f257de-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.597997 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.597970 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:17.706765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.706696 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 22:00:17.710794 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710771 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.710922 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710813 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.710922 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710840 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.710922 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710873 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.710922 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710910 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.711138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.710951 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.711138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.711030 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.711138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.711071 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.711284 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.711182 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\") pod \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\" (UID: \"f5180836-2929-4ef0-b4c5-36d20d1df3b5\") " Nov 03 22:00:17.711969 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.711919 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.712086 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.712041 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.712272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.712235 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.712368 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.712302 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.713542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.713512 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:17.713636 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.713582 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:17.713783 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.713753 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:17.714284 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.714263 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "kube-api-access-cv5hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:17.715493 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.715449 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-lxspc"] Nov 03 22:00:17.715562 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.715528 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy" pod "f5180836-2929-4ef0-b4c5-36d20d1df3b5" (UID: "f5180836-2929-4ef0-b4c5-36d20d1df3b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:17.812663 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812620 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812663 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812659 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812663 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812669 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812663 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812678 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv5hb\" (UniqueName: \"kubernetes.io/projected/f5180836-2929-4ef0-b4c5-36d20d1df3b5-kube-api-access-cv5hb\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812688 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812697 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812705 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812713 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:17.812921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:17.812722 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5180836-2929-4ef0-b4c5-36d20d1df3b5-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.344427 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.344391 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" path="/var/lib/kubelet/pods/55a0eeaa-c402-4ab8-ba8d-8f8db8784e64/volumes" Nov 03 22:00:18.344943 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.344922 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" path="/var/lib/kubelet/pods/c50978f4-39bd-4045-8c75-2ba431f257de/volumes" Nov 03 22:00:18.393568 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.393530 5973 generic.go:356] "Generic (PLEG): container finished" podID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" containerID="85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf" exitCode=0 Nov 03 22:00:18.393568 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.393566 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" event={"ID":"f5180836-2929-4ef0-b4c5-36d20d1df3b5","Type":"ContainerDied","Data":"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf"} Nov 03 22:00:18.393755 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.393588 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" Nov 03 22:00:18.393755 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.393627 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g" event={"ID":"f5180836-2929-4ef0-b4c5-36d20d1df3b5","Type":"ContainerDied","Data":"544de295431de65f281496f12b8eea5e0541055808af1beb180c585bea29a459"} Nov 03 22:00:18.393755 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.393648 5973 scope.go:122] "RemoveContainer" containerID="85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf" Nov 03 22:00:18.403294 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.403273 5973 scope.go:122] "RemoveContainer" containerID="85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf" Nov 03 22:00:18.403572 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:18.403552 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf\": container with ID starting with 85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf not found: ID does not exist" containerID="85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf" Nov 03 22:00:18.403629 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.403581 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf"} err="failed to get container status \"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf\": rpc error: code = NotFound desc = could not find container \"85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf\": container with ID starting with 85f94ba2653528bcfdcab83b2690df2443a93ae5d316733c998c26d590382ecf not found: ID does not exist" Nov 03 22:00:18.414958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.414935 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:18.418792 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:18.418772 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc16de-openshift-default-85fdcb545f-f2v2g"] Nov 03 22:00:20.344915 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:20.344883 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" path="/var/lib/kubelet/pods/f5180836-2929-4ef0-b4c5-36d20d1df3b5/volumes" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692341 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692927 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692946 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692967 5973 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerName="main" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692973 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerName="main" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.692992 5973 cpu_manager.go:414] "containerMap: removing container" podUID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693001 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693015 5973 cpu_manager.go:414] "containerMap: removing container" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693022 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693044 5973 cpu_manager.go:414] "containerMap: removing container" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693050 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693063 5973 cpu_manager.go:414] "containerMap: removing container" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693070 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693078 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693085 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693110 5973 cpu_manager.go:414] "containerMap: removing container" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693117 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693129 5973 cpu_manager.go:414] "containerMap: removing container" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693136 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693236 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693249 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ed1f9-7468-479c-87ef-b60024d86db4" containerName="main" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693258 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5180836-2929-4ef0-b4c5-36d20d1df3b5" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693265 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693274 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693281 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693289 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a0eeaa-c402-4ab8-ba8d-8f8db8784e64" containerName="cleanup" Nov 03 22:00:21.696838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.693298 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50978f4-39bd-4045-8c75-2ba431f257de" containerName="istio-proxy" Nov 03 22:00:21.699026 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.698712 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.710561 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.710536 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:21.751553 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751527 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751696 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751562 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751696 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751647 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751698 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751734 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751754 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751863 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751833 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751871 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.751957 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.751943 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bddc\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852781 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852751 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852933 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852794 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852933 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852827 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bddc\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852933 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852849 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852933 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852869 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.852933 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.852892 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853121 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853165 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853190 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853247 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853251 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853278 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853545 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.853840 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.853824 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.855415 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.855396 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.855677 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.855658 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.865424 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.865401 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bddc\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:21.865540 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:21.865475 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\") pod \"e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:22.018857 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.018775 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:22.307039 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.307009 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:22.307888 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:00:22.307863 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763e9389_6e8c_4d23_a8d4_521a80fada44.slice/crio-e80af9ae74fcb4213928651056cf1df4aaa1654667fb2d4d464de5d627db5a6d WatchSource:0}: Error finding container e80af9ae74fcb4213928651056cf1df4aaa1654667fb2d4d464de5d627db5a6d: Status 404 returned error can't find the container with id e80af9ae74fcb4213928651056cf1df4aaa1654667fb2d4d464de5d627db5a6d Nov 03 22:00:22.310071 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.310041 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:22.310144 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.310121 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:22.310188 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.310150 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:00:22.364619 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.364585 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:22.420670 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.420613 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" event={"ID":"763e9389-6e8c-4d23-a8d4-521a80fada44","Type":"ContainerStarted","Data":"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87"} Nov 03 22:00:22.420835 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.420707 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" event={"ID":"763e9389-6e8c-4d23-a8d4-521a80fada44","Type":"ContainerStarted","Data":"e80af9ae74fcb4213928651056cf1df4aaa1654667fb2d4d464de5d627db5a6d"} Nov 03 22:00:22.446402 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.446347 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" podStartSLOduration=1.446330006 podStartE2EDuration="1.446330006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:22.443441786 +0000 UTC m=+862.218291633" watchObservedRunningTime="2026-11-03 22:00:22.446330006 +0000 UTC m=+862.221179853" Nov 03 22:00:22.531684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.531645 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:22.635115 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.635030 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 22:00:22.635316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.635288 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" containerID="cri-o://079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15" gracePeriod=30 Nov 03 22:00:22.640982 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.640957 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:22.689365 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.689334 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:22.849196 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:22.849159 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:23.019294 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:23.019263 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:23.021048 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:23.021013 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:00:23.021181 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:23.021070 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:00:23.428015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:23.427982 5973 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" secret="" err="secret \"e2e-worker-w2-53a2cb-openshift-default-dockercfg-6hq7h\" not found" Nov 03 22:00:23.436865 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:23.436834 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:24.020003 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:24.019970 5973 patch_prober.go:29] interesting pod/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:00:24.020571 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:24.020023 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:00:24.434326 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:24.434291 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" containerID="cri-o://ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87" gracePeriod=30 Nov 03 22:00:27.015541 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:27.015508 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:27.205484 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:27.184778 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:28.244238 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.244152 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 22:00:28.349645 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349618 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349783 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349653 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349783 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349699 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349783 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349743 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349863 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349927 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349893 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349959 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349935 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.349994 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.349981 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.350049 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350015 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\") pod \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\" (UID: \"bdc9e252-a5b5-45b1-87d2-0e102d35f61a\") " Nov 03 22:00:28.350256 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350224 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:28.350474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350348 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:28.350474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350422 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:28.350614 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350351 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.350671 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350632 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:28.350671 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.350646 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:28.352912 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.352894 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:28.353263 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.353239 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:28.353321 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.353283 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "kube-api-access-2nwp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:28.353502 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.353478 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo" pod "bdc9e252-a5b5-45b1-87d2-0e102d35f61a" (UID: "bdc9e252-a5b5-45b1-87d2-0e102d35f61a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:28.452033 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.451985 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452033 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452020 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452033 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452036 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452050 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452064 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwp6\" (UniqueName: \"kubernetes.io/projected/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-kube-api-access-2nwp6\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452077 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452090 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.452309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.452104 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9e252-a5b5-45b1-87d2-0e102d35f61a-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:28.533043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.530699 5973 generic.go:356] "Generic (PLEG): container finished" podID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerID="079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15" exitCode=0 Nov 03 22:00:28.533043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.530943 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerDied","Data":"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15"} Nov 03 22:00:28.533043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.530996 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" event={"ID":"bdc9e252-a5b5-45b1-87d2-0e102d35f61a","Type":"ContainerDied","Data":"15f4d51cd439ef285d63cb49646fcb54ac11bfe00cc7d77805b047a26c20419a"} Nov 03 22:00:28.533043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.531028 5973 scope.go:122] "RemoveContainer" containerID="079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15" Nov 03 22:00:28.533043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.531198 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl" Nov 03 22:00:28.545935 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.545904 5973 scope.go:122] "RemoveContainer" containerID="0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762" Nov 03 22:00:28.555151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.555133 5973 scope.go:122] "RemoveContainer" containerID="079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15" Nov 03 22:00:28.555427 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:28.555405 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15\": container with ID starting with 079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15 not found: ID does not exist" containerID="079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15" Nov 03 22:00:28.555518 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.555436 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15"} err="failed to get container status \"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15\": rpc error: code = NotFound desc = could not find container \"079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15\": container with ID starting with 079b488a7053328d99b587fe62539ab8cbd29c4eb0cb9927327bfcaf35dfbf15 not found: ID does not exist" Nov 03 22:00:28.584496 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.571484 5973 scope.go:122] "RemoveContainer" containerID="0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762" Nov 03 22:00:28.584496 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:28.575882 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762\": container with ID starting with 0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762 not found: ID does not exist" containerID="0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762" Nov 03 22:00:28.584496 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.575924 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762"} err="failed to get container status \"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762\": rpc error: code = NotFound desc = could not find container \"0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762\": container with ID starting with 0c8138ebf19ead210d46256feb752f672c634790e2a7407eb95e7597920e9762 not found: ID does not exist" Nov 03 22:00:28.603496 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.602515 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 22:00:28.632183 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.632135 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-wczsl"] Nov 03 22:00:28.676348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:28.676283 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:29.113816 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.113772 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:29.680670 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.680642 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:29.766515 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766418 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766515 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766482 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766526 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766598 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766626 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766673 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766724 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:29.766928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766743 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766795 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.766928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766864 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\") pod \"763e9389-6e8c-4d23-a8d4-521a80fada44\" (UID: \"763e9389-6e8c-4d23-a8d4-521a80fada44\") " Nov 03 22:00:29.767107 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766923 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:29.767107 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.766962 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:29.767107 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767097 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:29.767254 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767161 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:29.767254 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767242 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.767360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767260 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.767360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767275 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.767360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767287 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763e9389-6e8c-4d23-a8d4-521a80fada44-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.767360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.767295 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.768879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.768848 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "kube-api-access-6bddc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:29.769141 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.769113 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:29.769141 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.769125 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:00:29.769293 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.769224 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo" pod "763e9389-6e8c-4d23-a8d4-521a80fada44" (UID: "763e9389-6e8c-4d23-a8d4-521a80fada44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:00:29.868757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.868727 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bddc\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-kube-api-access-6bddc\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.868757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.868752 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.868757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.868762 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:29.868975 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:29.868771 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763e9389-6e8c-4d23-a8d4-521a80fada44-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:30.346121 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.346087 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" path="/var/lib/kubelet/pods/bdc9e252-a5b5-45b1-87d2-0e102d35f61a/volumes" Nov 03 22:00:30.545261 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.545226 5973 generic.go:356] "Generic (PLEG): container finished" podID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerID="ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87" exitCode=0 Nov 03 22:00:30.545407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.545282 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" event={"ID":"763e9389-6e8c-4d23-a8d4-521a80fada44","Type":"ContainerDied","Data":"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87"} Nov 03 22:00:30.545407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.545321 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" event={"ID":"763e9389-6e8c-4d23-a8d4-521a80fada44","Type":"ContainerDied","Data":"e80af9ae74fcb4213928651056cf1df4aaa1654667fb2d4d464de5d627db5a6d"} Nov 03 22:00:30.545407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.545341 5973 scope.go:122] "RemoveContainer" containerID="ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87" Nov 03 22:00:30.545407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.545283 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r" Nov 03 22:00:30.554970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.554950 5973 scope.go:122] "RemoveContainer" containerID="ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87" Nov 03 22:00:30.555276 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:30.555258 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87\": container with ID starting with ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87 not found: ID does not exist" containerID="ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87" Nov 03 22:00:30.555327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.555286 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87"} err="failed to get container status \"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87\": rpc error: code = NotFound desc = could not find container \"ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87\": container with ID starting with ef6ba5f05ec274b92d9b1be1c909da8f93060ffbc7724bd2e03bb92ffaca9c87 not found: ID does not exist" Nov 03 22:00:30.566785 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.566762 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:30.572692 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:30.572669 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-53a2cb-openshift-default-7fc9bd965c-5pr4r"] Nov 03 22:00:32.344970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:32.344938 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" path="/var/lib/kubelet/pods/763e9389-6e8c-4d23-a8d4-521a80fada44/volumes" Nov 03 22:00:34.422760 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:34.422729 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:44.398810 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:00:44.398770 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:00:47.607229 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607194 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz"] Nov 03 22:00:47.607803 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607784 5973 cpu_manager.go:414] "containerMap: removing container" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" Nov 03 22:00:47.607879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607806 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" Nov 03 22:00:47.607879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607848 5973 cpu_manager.go:414] "containerMap: removing container" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:00:47.607879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607856 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:00:47.608034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607968 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:00:47.608034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607982 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:00:47.608034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.607993 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e508779-36bd-4777-aa76-beffba2b7c0d" containerName="cleanup" Nov 03 22:00:47.608034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.608003 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="763e9389-6e8c-4d23-a8d4-521a80fada44" containerName="istio-proxy" Nov 03 22:00:47.611097 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.611078 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.618234 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.618056 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:00:47.618234 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.618099 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:00:47.618359 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.618254 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:00:47.618359 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.618335 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:00:47.622204 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.622183 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz"] Nov 03 22:00:47.729649 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.729623 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntxv\" (UniqueName: \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.729786 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.729687 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.729850 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.729797 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.830995 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.830964 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.831122 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.831016 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntxv\" (UniqueName: \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.831122 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.831084 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.831745 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.831721 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.833533 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.833510 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.844077 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.844057 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntxv\" (UniqueName: \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\") pod \"maas-api-revoke-sub-85237e45-d4xkz\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:47.925132 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:47.925111 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:48.209575 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:48.209011 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz"] Nov 03 22:00:48.655173 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:48.655136 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" event={"ID":"8a3e832f-5874-419a-a24b-0156065e42b6","Type":"ContainerStarted","Data":"df2293cca2c05c277446c2eff95d358736485af083e18248af699c1a364e1025"} Nov 03 22:00:48.655698 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:48.655187 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" event={"ID":"8a3e832f-5874-419a-a24b-0156065e42b6","Type":"ContainerStarted","Data":"fa1fee15f579431bb56d9b82e1baedeaf331fe0c280b240cc72bc3340813f4d4"} Nov 03 22:00:48.675684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:48.675637 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" podStartSLOduration=1.675624763 podStartE2EDuration="1.675624763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:48.672974534 +0000 UTC m=+888.447824381" watchObservedRunningTime="2026-11-03 22:00:48.675624763 +0000 UTC m=+888.450474646" Nov 03 22:00:49.662825 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:49.662791 5973 generic.go:356] "Generic (PLEG): container finished" podID="8a3e832f-5874-419a-a24b-0156065e42b6" containerID="df2293cca2c05c277446c2eff95d358736485af083e18248af699c1a364e1025" exitCode=0 Nov 03 22:00:49.663239 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:49.662873 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" event={"ID":"8a3e832f-5874-419a-a24b-0156065e42b6","Type":"ContainerDied","Data":"df2293cca2c05c277446c2eff95d358736485af083e18248af699c1a364e1025"} Nov 03 22:00:50.804079 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.804057 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:50.960562 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.960478 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\") pod \"8a3e832f-5874-419a-a24b-0156065e42b6\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " Nov 03 22:00:50.960562 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.960527 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\" (UniqueName: \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\") pod \"8a3e832f-5874-419a-a24b-0156065e42b6\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " Nov 03 22:00:50.960806 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.960619 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\") pod \"8a3e832f-5874-419a-a24b-0156065e42b6\" (UID: \"8a3e832f-5874-419a-a24b-0156065e42b6\") " Nov 03 22:00:50.960806 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.960778 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca" pod "8a3e832f-5874-419a-a24b-0156065e42b6" (UID: "8a3e832f-5874-419a-a24b-0156065e42b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:50.962794 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.962770 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv" pod "8a3e832f-5874-419a-a24b-0156065e42b6" (UID: "8a3e832f-5874-419a-a24b-0156065e42b6"). InnerVolumeSpecName "kube-api-access-tntxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:50.962876 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:50.962855 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca" pod "8a3e832f-5874-419a-a24b-0156065e42b6" (UID: "8a3e832f-5874-419a-a24b-0156065e42b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:51.061606 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.061581 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:51.061606 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.061603 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntxv\" (UniqueName: \"kubernetes.io/projected/8a3e832f-5874-419a-a24b-0156065e42b6-kube-api-access-tntxv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:51.061740 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.061612 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3e832f-5874-419a-a24b-0156065e42b6-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:51.676867 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.676837 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" Nov 03 22:00:51.677078 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.676839 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz" event={"ID":"8a3e832f-5874-419a-a24b-0156065e42b6","Type":"ContainerDied","Data":"fa1fee15f579431bb56d9b82e1baedeaf331fe0c280b240cc72bc3340813f4d4"} Nov 03 22:00:51.677078 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:51.676962 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa1fee15f579431bb56d9b82e1baedeaf331fe0c280b240cc72bc3340813f4d4" Nov 03 22:00:56.504732 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:56.504690 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:59.124501 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:00:59.124440 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:01.010814 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:01.010782 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:14.398845 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:01:14.398762 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:01:29.245514 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:29.245480 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:01:31.608613 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:31.608570 5973 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-5cffb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:31.609250 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:31.609214 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-5cffb" podUID="696e0ecd-a455-4af2-bdb4-9932fab6d32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:34.207756 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.207728 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8"] Nov 03 22:01:34.208137 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.208092 5973 cpu_manager.go:414] "containerMap: removing container" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:01:34.208137 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.208103 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc9e252-a5b5-45b1-87d2-0e102d35f61a" containerName="istio-proxy" Nov 03 22:01:34.208137 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.208123 5973 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e832f-5874-419a-a24b-0156065e42b6" containerName="revoke-keys" Nov 03 22:01:34.208137 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.208129 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e832f-5874-419a-a24b-0156065e42b6" containerName="revoke-keys" Nov 03 22:01:34.208269 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.208195 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3e832f-5874-419a-a24b-0156065e42b6" containerName="revoke-keys" Nov 03 22:01:34.211666 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.211645 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.217761 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.217734 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:01:34.217936 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.217820 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:01:34.219022 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.219003 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:01:34.219156 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.219097 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:01:34.221995 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.221976 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8"] Nov 03 22:01:34.263256 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.263223 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2cm\" (UniqueName: \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.263430 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.263356 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.263430 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.263398 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.364328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.364293 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.364328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.364336 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.364563 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.364362 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2cm\" (UniqueName: \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.364946 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.364922 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.366879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.366857 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.376754 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.376727 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2cm\" (UniqueName: \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\") pod \"maas-api-revoke-sub-c1e271a9-5xhf8\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.528146 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.528065 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:34.807071 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.807042 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8"] Nov 03 22:01:34.808520 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:01:34.808487 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2db19b0_9030_4ded_9981_d88032c92670.slice/crio-c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1 WatchSource:0}: Error finding container c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1: Status 404 returned error can't find the container with id c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1 Nov 03 22:01:34.955073 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:34.955041 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" event={"ID":"a2db19b0-9030-4ded-9981-d88032c92670","Type":"ContainerStarted","Data":"c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1"} Nov 03 22:01:35.962271 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:35.962240 5973 generic.go:356] "Generic (PLEG): container finished" podID="a2db19b0-9030-4ded-9981-d88032c92670" containerID="c2b45eaf00bd8d595b29494726dbfca2f625f67aeaeac39e3d4c9a19ce38b7dd" exitCode=0 Nov 03 22:01:35.962659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:35.962337 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" event={"ID":"a2db19b0-9030-4ded-9981-d88032c92670","Type":"ContainerDied","Data":"c2b45eaf00bd8d595b29494726dbfca2f625f67aeaeac39e3d4c9a19ce38b7dd"} Nov 03 22:01:37.121865 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.121839 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:37.189700 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.189672 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\" (UniqueName: \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\") pod \"a2db19b0-9030-4ded-9981-d88032c92670\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " Nov 03 22:01:37.189878 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.189733 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\") pod \"a2db19b0-9030-4ded-9981-d88032c92670\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " Nov 03 22:01:37.189878 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.189839 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\") pod \"a2db19b0-9030-4ded-9981-d88032c92670\" (UID: \"a2db19b0-9030-4ded-9981-d88032c92670\") " Nov 03 22:01:37.190177 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.190145 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca" pod "a2db19b0-9030-4ded-9981-d88032c92670" (UID: "a2db19b0-9030-4ded-9981-d88032c92670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:01:37.191792 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.191768 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm" pod "a2db19b0-9030-4ded-9981-d88032c92670" (UID: "a2db19b0-9030-4ded-9981-d88032c92670"). InnerVolumeSpecName "kube-api-access-xn2cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:01:37.191876 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.191845 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca" pod "a2db19b0-9030-4ded-9981-d88032c92670" (UID: "a2db19b0-9030-4ded-9981-d88032c92670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:01:37.291534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.291441 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:37.291534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.291486 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2cm\" (UniqueName: \"kubernetes.io/projected/a2db19b0-9030-4ded-9981-d88032c92670-kube-api-access-xn2cm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:37.291534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.291497 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2db19b0-9030-4ded-9981-d88032c92670-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:37.975777 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.975750 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" Nov 03 22:01:37.975928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.975748 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8" event={"ID":"a2db19b0-9030-4ded-9981-d88032c92670","Type":"ContainerDied","Data":"c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1"} Nov 03 22:01:37.975928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.975904 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26ea20e71b25be9a58a8d457280da2bc81757413bcb72f0191290a3526f9cd1" Nov 03 22:01:37.994817 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:37.994783 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:38.150990 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:38.150961 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:38.312659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:38.312572 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:44.399248 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:01:44.399210 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:01:45.028500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:45.028443 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:01:48.886703 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:48.886664 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:01:52.927055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.927022 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk"] Nov 03 22:01:52.927623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.927604 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a2db19b0-9030-4ded-9981-d88032c92670" containerName="revoke-keys" Nov 03 22:01:52.927717 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.927625 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2db19b0-9030-4ded-9981-d88032c92670" containerName="revoke-keys" Nov 03 22:01:52.927774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.927760 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2db19b0-9030-4ded-9981-d88032c92670" containerName="revoke-keys" Nov 03 22:01:52.936857 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.936419 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:52.941151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.941124 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk"] Nov 03 22:01:52.941364 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.941341 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:01:52.942415 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.942389 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:01:52.942541 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.942491 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:01:52.942586 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:52.942553 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:01:53.065980 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.065944 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.065980 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.065977 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwfc\" (UniqueName: \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.066194 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.066160 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.167011 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.166982 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.167179 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.167020 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.167179 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.167045 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwfc\" (UniqueName: \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.167843 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.167812 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.169625 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.169599 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.180100 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.180042 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwfc\" (UniqueName: \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\") pod \"maas-api-revoke-sub-2e1f5b5f-gqlzk\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.251483 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.251436 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:53.527289 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:53.527252 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk"] Nov 03 22:01:53.528161 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:01:53.528138 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce4a12_34da_4a25_bf3e_d09ffa6e7b4b.slice/crio-5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387 WatchSource:0}: Error finding container 5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387: Status 404 returned error can't find the container with id 5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387 Nov 03 22:01:54.079160 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:54.079124 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" event={"ID":"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b","Type":"ContainerStarted","Data":"3f776ee15042a3eb3a4684ce8ad6c1e794b25f32c9da739303d0be509678d3a3"} Nov 03 22:01:54.079592 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:54.079183 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" event={"ID":"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b","Type":"ContainerStarted","Data":"5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387"} Nov 03 22:01:54.098542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:54.098489 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" podStartSLOduration=2.098476219 podStartE2EDuration="2.098476219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:54.097028427 +0000 UTC m=+953.871878277" watchObservedRunningTime="2026-11-03 22:01:54.098476219 +0000 UTC m=+953.873326067" Nov 03 22:01:55.088086 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:55.088051 5973 generic.go:356] "Generic (PLEG): container finished" podID="a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" containerID="3f776ee15042a3eb3a4684ce8ad6c1e794b25f32c9da739303d0be509678d3a3" exitCode=0 Nov 03 22:01:55.088531 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:55.088136 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" event={"ID":"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b","Type":"ContainerDied","Data":"3f776ee15042a3eb3a4684ce8ad6c1e794b25f32c9da739303d0be509678d3a3"} Nov 03 22:01:56.225091 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.225069 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:56.294590 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.294554 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\") pod \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " Nov 03 22:01:56.294780 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.294667 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\") pod \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " Nov 03 22:01:56.294780 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.294717 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\" (UniqueName: \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\") pod \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\" (UID: \"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b\") " Nov 03 22:01:56.295008 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.294985 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca" pod "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" (UID: "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:01:56.296718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.296695 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca" pod "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" (UID: "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:01:56.296820 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.296763 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc" pod "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" (UID: "a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b"). InnerVolumeSpecName "kube-api-access-kdwfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:01:56.395827 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.395798 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwfc\" (UniqueName: \"kubernetes.io/projected/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-kube-api-access-kdwfc\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:56.395827 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.395824 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:56.395991 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.395835 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:01:56.552832 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:56.552795 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:57.101552 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:57.101521 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" Nov 03 22:01:57.101552 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:57.101546 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk" event={"ID":"a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b","Type":"ContainerDied","Data":"5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387"} Nov 03 22:01:57.101762 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:01:57.101628 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3be12bfc5def1e366788de491af69b40062e490d117f9d8f5f03006e44d387" Nov 03 22:02:03.651639 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.651607 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:02:03.912482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.912389 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697"] Nov 03 22:02:03.912872 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.912858 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" containerName="revoke-keys" Nov 03 22:02:03.912915 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.912874 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" containerName="revoke-keys" Nov 03 22:02:03.912968 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.912959 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" containerName="revoke-keys" Nov 03 22:02:03.916072 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.916054 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:03.921059 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.921023 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:02:03.922230 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.922128 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:02:03.923970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.923949 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:02:03.924090 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.924067 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:02:03.925271 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.925241 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697"] Nov 03 22:02:03.967579 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.967539 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:03.967750 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.967609 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:03.967750 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:03.967695 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4hr\" (UniqueName: \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.068865 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.068812 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.069101 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.068884 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4hr\" (UniqueName: \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.069101 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.069020 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.069816 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.069786 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.073323 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.073287 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.082832 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.082804 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4hr\" (UniqueName: \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\") pod \"maas-api-revoke-sub-ec843018-f8697\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.231902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.231824 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:04.508690 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:04.508663 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697"] Nov 03 22:02:04.509568 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:02:04.509541 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd06096_8acb_4d3c_8cf5_4fc8c2854fdb.slice/crio-a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80 WatchSource:0}: Error finding container a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80: Status 404 returned error can't find the container with id a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80 Nov 03 22:02:05.154020 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:05.153991 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" event={"ID":"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb","Type":"ContainerStarted","Data":"8b2db8dde955de1fdca1b7ff9d771a9ebcae5b70df627f8a53656f996f33b4e4"} Nov 03 22:02:05.154510 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:05.154037 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" event={"ID":"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb","Type":"ContainerStarted","Data":"a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80"} Nov 03 22:02:05.174786 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:05.174742 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" podStartSLOduration=2.174728107 podStartE2EDuration="2.174728107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:05.172349272 +0000 UTC m=+964.947199119" watchObservedRunningTime="2026-11-03 22:02:05.174728107 +0000 UTC m=+964.949578001" Nov 03 22:02:06.164535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:06.164496 5973 generic.go:356] "Generic (PLEG): container finished" podID="6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" containerID="8b2db8dde955de1fdca1b7ff9d771a9ebcae5b70df627f8a53656f996f33b4e4" exitCode=0 Nov 03 22:02:06.164955 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:06.164552 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" event={"ID":"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb","Type":"ContainerDied","Data":"8b2db8dde955de1fdca1b7ff9d771a9ebcae5b70df627f8a53656f996f33b4e4"} Nov 03 22:02:07.646310 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.646271 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:07.736886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.736839 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\") pod \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " Nov 03 22:02:07.737219 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.737198 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\" (UniqueName: \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\") pod \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " Nov 03 22:02:07.737793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.737346 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca" pod "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" (UID: "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:07.738215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.738196 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\") pod \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\" (UID: \"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb\") " Nov 03 22:02:07.738712 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.738688 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:07.740631 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.740604 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca" pod "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" (UID: "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:07.740745 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.740724 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr" pod "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" (UID: "6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb"). InnerVolumeSpecName "kube-api-access-4n4hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:07.840293 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.840258 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4hr\" (UniqueName: \"kubernetes.io/projected/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-kube-api-access-4n4hr\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:07.840293 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:07.840288 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:08.247223 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:08.247194 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" Nov 03 22:02:08.247402 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:08.247244 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697" event={"ID":"6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb","Type":"ContainerDied","Data":"a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80"} Nov 03 22:02:08.247402 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:08.247322 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a873d244a7078eb2e864e582d908728c14e09ab7153c2430bc565d0ac80f80" Nov 03 22:02:14.399867 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:02:14.399829 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:02:16.420292 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:16.420254 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:02:24.243575 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:24.243538 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:24.398296 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:24.398236 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:29.932235 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.932204 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw"] Nov 03 22:02:29.932833 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.932813 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" containerName="revoke-keys" Nov 03 22:02:29.932914 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.932837 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" containerName="revoke-keys" Nov 03 22:02:29.932965 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.932945 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" containerName="revoke-keys" Nov 03 22:02:29.936275 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.936246 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:29.943570 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.943540 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:02:29.943570 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.943550 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:02:29.947568 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.947540 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw"] Nov 03 22:02:29.956759 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.956735 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:02:29.957062 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:29.956768 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:02:30.053980 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.053944 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.053980 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.053982 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29qv\" (UniqueName: \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.054258 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.054134 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.155218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.155185 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.155218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.155223 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z29qv\" (UniqueName: \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.155442 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.155285 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.156004 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.155984 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.157666 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.157647 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.167674 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.167654 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29qv\" (UniqueName: \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\") pod \"maas-api-revoke-sub-046e1343-l4tqw\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.251175 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.251107 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:30.530886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:30.530857 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw"] Nov 03 22:02:30.531772 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:02:30.531741 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf541d42_f385_4e51_9788_265dc68719d2.slice/crio-af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e WatchSource:0}: Error finding container af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e: Status 404 returned error can't find the container with id af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e Nov 03 22:02:31.419529 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:31.419486 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" event={"ID":"cf541d42-f385-4e51-9788-265dc68719d2","Type":"ContainerStarted","Data":"d4d85fb266ed4b3de01b95410d8561643fe5caab482fddb7fdd01d7076325e98"} Nov 03 22:02:31.419901 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:31.419551 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" event={"ID":"cf541d42-f385-4e51-9788-265dc68719d2","Type":"ContainerStarted","Data":"af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e"} Nov 03 22:02:31.440337 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:31.440291 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" podStartSLOduration=2.440277363 podStartE2EDuration="2.440277363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:31.438850555 +0000 UTC m=+991.213700403" watchObservedRunningTime="2026-11-03 22:02:31.440277363 +0000 UTC m=+991.215127210" Nov 03 22:02:32.426960 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:32.426925 5973 generic.go:356] "Generic (PLEG): container finished" podID="cf541d42-f385-4e51-9788-265dc68719d2" containerID="d4d85fb266ed4b3de01b95410d8561643fe5caab482fddb7fdd01d7076325e98" exitCode=0 Nov 03 22:02:32.427404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:32.427019 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" event={"ID":"cf541d42-f385-4e51-9788-265dc68719d2","Type":"ContainerDied","Data":"d4d85fb266ed4b3de01b95410d8561643fe5caab482fddb7fdd01d7076325e98"} Nov 03 22:02:33.567921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.567896 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:33.688006 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.687975 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\") pod \"cf541d42-f385-4e51-9788-265dc68719d2\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " Nov 03 22:02:33.688218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.688068 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\" (UniqueName: \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\") pod \"cf541d42-f385-4e51-9788-265dc68719d2\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " Nov 03 22:02:33.688218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.688112 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\") pod \"cf541d42-f385-4e51-9788-265dc68719d2\" (UID: \"cf541d42-f385-4e51-9788-265dc68719d2\") " Nov 03 22:02:33.688577 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.688552 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca" pod "cf541d42-f385-4e51-9788-265dc68719d2" (UID: "cf541d42-f385-4e51-9788-265dc68719d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:33.690325 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.690304 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca" pod "cf541d42-f385-4e51-9788-265dc68719d2" (UID: "cf541d42-f385-4e51-9788-265dc68719d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:33.690391 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.690371 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv" pod "cf541d42-f385-4e51-9788-265dc68719d2" (UID: "cf541d42-f385-4e51-9788-265dc68719d2"). InnerVolumeSpecName "kube-api-access-z29qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:33.789023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.788941 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:33.789023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.788973 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf541d42-f385-4e51-9788-265dc68719d2-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:33.789023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:33.788983 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z29qv\" (UniqueName: \"kubernetes.io/projected/cf541d42-f385-4e51-9788-265dc68719d2-kube-api-access-z29qv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:34.441114 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:34.441080 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" Nov 03 22:02:34.441114 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:34.441095 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw" event={"ID":"cf541d42-f385-4e51-9788-265dc68719d2","Type":"ContainerDied","Data":"af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e"} Nov 03 22:02:34.441405 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:34.441157 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0351e4947915e34071131e36659b3a2526c69e37608d2329617430a6d3bf1e" Nov 03 22:02:40.133414 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.133377 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4"] Nov 03 22:02:40.133834 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.133820 5973 cpu_manager.go:414] "containerMap: removing container" podUID="cf541d42-f385-4e51-9788-265dc68719d2" containerName="revoke-keys" Nov 03 22:02:40.133876 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.133836 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf541d42-f385-4e51-9788-265dc68719d2" containerName="revoke-keys" Nov 03 22:02:40.133943 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.133934 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf541d42-f385-4e51-9788-265dc68719d2" containerName="revoke-keys" Nov 03 22:02:40.137378 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.137361 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.142429 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.142363 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:02:40.143269 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.143248 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:02:40.145476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.143498 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:02:40.145476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.143619 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:02:40.147385 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.147363 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4"] Nov 03 22:02:40.250385 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.250288 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf248\" (UniqueName: \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.250573 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.250396 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.250573 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.250473 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.351838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.351802 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf248\" (UniqueName: \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.352007 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.351865 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.352007 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.351925 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.352776 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.352703 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.354419 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.354400 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.363845 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.363821 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf248\" (UniqueName: \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\") pod \"maas-api-revoke-sub-d3fba056-58hn4\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.455200 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.455159 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:40.634231 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.634194 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:02:40.738061 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:40.738036 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4"] Nov 03 22:02:40.739162 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:02:40.739136 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e185d7f_b2cd_433f_b6f1_d0b506966e3e.slice/crio-dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a WatchSource:0}: Error finding container dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a: Status 404 returned error can't find the container with id dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a Nov 03 22:02:41.489771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:41.489728 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" event={"ID":"4e185d7f-b2cd-433f-b6f1-d0b506966e3e","Type":"ContainerStarted","Data":"5e306dfc17922f1bee0ab4b60d02e8bc1c123213fcf7c45ec2c0657c81b4a9e2"} Nov 03 22:02:41.490325 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:41.489802 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" event={"ID":"4e185d7f-b2cd-433f-b6f1-d0b506966e3e","Type":"ContainerStarted","Data":"dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a"} Nov 03 22:02:41.510524 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:41.509996 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" podStartSLOduration=1.5099799059999999 podStartE2EDuration="1.509979906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:41.5077593 +0000 UTC m=+1001.282609148" watchObservedRunningTime="2026-11-03 22:02:41.509979906 +0000 UTC m=+1001.284829758" Nov 03 22:02:42.518487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:42.517582 5973 generic.go:356] "Generic (PLEG): container finished" podID="4e185d7f-b2cd-433f-b6f1-d0b506966e3e" containerID="5e306dfc17922f1bee0ab4b60d02e8bc1c123213fcf7c45ec2c0657c81b4a9e2" exitCode=0 Nov 03 22:02:42.518487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:42.517678 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" event={"ID":"4e185d7f-b2cd-433f-b6f1-d0b506966e3e","Type":"ContainerDied","Data":"5e306dfc17922f1bee0ab4b60d02e8bc1c123213fcf7c45ec2c0657c81b4a9e2"} Nov 03 22:02:43.663757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.663732 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:43.825905 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.825817 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\" (UniqueName: \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\") pod \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " Nov 03 22:02:43.826058 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.825956 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\") pod \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " Nov 03 22:02:43.826058 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.825996 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\") pod \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\" (UID: \"4e185d7f-b2cd-433f-b6f1-d0b506966e3e\") " Nov 03 22:02:43.826344 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.826320 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca" pod "4e185d7f-b2cd-433f-b6f1-d0b506966e3e" (UID: "4e185d7f-b2cd-433f-b6f1-d0b506966e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:43.828176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.828154 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248" pod "4e185d7f-b2cd-433f-b6f1-d0b506966e3e" (UID: "4e185d7f-b2cd-433f-b6f1-d0b506966e3e"). InnerVolumeSpecName "kube-api-access-zf248". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:43.828257 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.828229 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca" pod "4e185d7f-b2cd-433f-b6f1-d0b506966e3e" (UID: "4e185d7f-b2cd-433f-b6f1-d0b506966e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:43.927169 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.927137 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf248\" (UniqueName: \"kubernetes.io/projected/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-kube-api-access-zf248\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:43.927169 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.927162 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:43.927169 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:43.927171 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e185d7f-b2cd-433f-b6f1-d0b506966e3e-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:44.398631 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:02:44.398592 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:02:44.532702 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:44.532668 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" event={"ID":"4e185d7f-b2cd-433f-b6f1-d0b506966e3e","Type":"ContainerDied","Data":"dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a"} Nov 03 22:02:44.532702 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:44.532689 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4" Nov 03 22:02:44.532918 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:44.532715 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc6402dbf698d51b397837e66033366b66d7265ce06c542becb029a50b6b3f8a" Nov 03 22:02:50.846793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:50.846761 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:50.933534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:50.933500 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:02:55.575719 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:55.575686 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:02:56.604894 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:02:56.604855 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:00.778367 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.778329 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9"] Nov 03 22:03:00.778959 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.778938 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4e185d7f-b2cd-433f-b6f1-d0b506966e3e" containerName="revoke-keys" Nov 03 22:03:00.779036 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.778962 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e185d7f-b2cd-433f-b6f1-d0b506966e3e" containerName="revoke-keys" Nov 03 22:03:00.779091 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.779068 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e185d7f-b2cd-433f-b6f1-d0b506966e3e" containerName="revoke-keys" Nov 03 22:03:00.782369 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.782345 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.784046 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.784018 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.784178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.784081 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwfv\" (UniqueName: \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.784246 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.784195 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.788731 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.788707 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:03:00.788853 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.788801 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:03:00.789284 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.789251 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:03:00.789374 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.789310 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:03:00.793248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.793221 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9"] Nov 03 22:03:00.884915 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.884875 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.884915 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.884924 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwfv\" (UniqueName: \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.885162 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.884967 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.885586 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.885562 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.887357 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.887336 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:00.897324 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:00.897303 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwfv\" (UniqueName: \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\") pod \"maas-api-revoke-sub-ae594979-lgnt9\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:01.100338 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:01.100253 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:01.378377 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:01.378345 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9"] Nov 03 22:03:01.379003 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:03:01.378976 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd272b7a_d9ea_4fa5_9e31_6240195b1ace.slice/crio-3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117 WatchSource:0}: Error finding container 3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117: Status 404 returned error can't find the container with id 3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117 Nov 03 22:03:01.662537 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:01.662495 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" event={"ID":"fd272b7a-d9ea-4fa5-9e31-6240195b1ace","Type":"ContainerStarted","Data":"3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117"} Nov 03 22:03:02.670494 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:02.670445 5973 generic.go:356] "Generic (PLEG): container finished" podID="fd272b7a-d9ea-4fa5-9e31-6240195b1ace" containerID="afae7a3ab57fbbb05ef3bf9ee41e6b032738d766e6c8359145b31f942d31a78d" exitCode=0 Nov 03 22:03:02.670932 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:02.670527 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" event={"ID":"fd272b7a-d9ea-4fa5-9e31-6240195b1ace","Type":"ContainerDied","Data":"afae7a3ab57fbbb05ef3bf9ee41e6b032738d766e6c8359145b31f942d31a78d"} Nov 03 22:03:03.815096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.815064 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:03.910613 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.910580 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\") pod \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " Nov 03 22:03:03.910782 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.910664 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\" (UniqueName: \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\") pod \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " Nov 03 22:03:03.910782 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.910689 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\") pod \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\" (UID: \"fd272b7a-d9ea-4fa5-9e31-6240195b1ace\") " Nov 03 22:03:03.911015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.910987 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca" pod "fd272b7a-d9ea-4fa5-9e31-6240195b1ace" (UID: "fd272b7a-d9ea-4fa5-9e31-6240195b1ace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:03.912909 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.912884 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv" pod "fd272b7a-d9ea-4fa5-9e31-6240195b1ace" (UID: "fd272b7a-d9ea-4fa5-9e31-6240195b1ace"). InnerVolumeSpecName "kube-api-access-tlwfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:03.913016 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:03.912951 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca" pod "fd272b7a-d9ea-4fa5-9e31-6240195b1ace" (UID: "fd272b7a-d9ea-4fa5-9e31-6240195b1ace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:04.011448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.011369 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlwfv\" (UniqueName: \"kubernetes.io/projected/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-kube-api-access-tlwfv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:04.011448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.011397 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:04.011448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.011407 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd272b7a-d9ea-4fa5-9e31-6240195b1ace-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:04.684308 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.684278 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" Nov 03 22:03:04.684308 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.684311 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9" event={"ID":"fd272b7a-d9ea-4fa5-9e31-6240195b1ace","Type":"ContainerDied","Data":"3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117"} Nov 03 22:03:04.684579 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:04.684358 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b233a7347430d90ceb3fbca7589b67a6ca35c5ff11fea837aaf12d9dddad117" Nov 03 22:03:11.529248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:11.529212 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:14.401520 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:03:14.401441 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:03:15.313482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:15.313127 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:18.963030 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.962991 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm"] Nov 03 22:03:18.963545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.963385 5973 cpu_manager.go:414] "containerMap: removing container" podUID="fd272b7a-d9ea-4fa5-9e31-6240195b1ace" containerName="revoke-keys" Nov 03 22:03:18.963545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.963396 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd272b7a-d9ea-4fa5-9e31-6240195b1ace" containerName="revoke-keys" Nov 03 22:03:18.963545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.963452 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd272b7a-d9ea-4fa5-9e31-6240195b1ace" containerName="revoke-keys" Nov 03 22:03:18.966818 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.966802 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:18.973486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.972960 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:03:18.973486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.973047 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:03:18.973486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.973113 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:03:18.973486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.973139 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:03:18.983000 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:18.976868 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm"] Nov 03 22:03:19.063838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.063804 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.064008 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.063871 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrnt\" (UniqueName: \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.064008 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.063954 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.164637 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.164600 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.164835 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.164698 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrnt\" (UniqueName: \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.164835 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.164731 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.165246 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.165221 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.167426 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.167398 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.177685 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.177663 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrnt\" (UniqueName: \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\") pod \"maas-api-revoke-sub-a4aef7d7-4hkxm\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.287286 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.287204 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:19.563544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.563513 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm"] Nov 03 22:03:19.564570 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:03:19.564545 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf87ea2e_03ca_4f7e_8f08_0107074dd1ed.slice/crio-5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180 WatchSource:0}: Error finding container 5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180: Status 404 returned error can't find the container with id 5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180 Nov 03 22:03:19.785448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:19.785417 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" event={"ID":"df87ea2e-03ca-4f7e-8f08-0107074dd1ed","Type":"ContainerStarted","Data":"5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180"} Nov 03 22:03:20.792861 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:20.792823 5973 generic.go:356] "Generic (PLEG): container finished" podID="df87ea2e-03ca-4f7e-8f08-0107074dd1ed" containerID="207898bb81e5a0714a6a9812dcd9795f79d18e94f7840aac3ea561eb687bb2b4" exitCode=0 Nov 03 22:03:20.793275 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:20.792874 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" event={"ID":"df87ea2e-03ca-4f7e-8f08-0107074dd1ed","Type":"ContainerDied","Data":"207898bb81e5a0714a6a9812dcd9795f79d18e94f7840aac3ea561eb687bb2b4"} Nov 03 22:03:21.930534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.930504 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:21.991931 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.991898 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\") pod \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " Nov 03 22:03:21.992099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.991952 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\") pod \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " Nov 03 22:03:21.992099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.992030 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\" (UniqueName: \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\") pod \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\" (UID: \"df87ea2e-03ca-4f7e-8f08-0107074dd1ed\") " Nov 03 22:03:21.992381 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.992352 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca" pod "df87ea2e-03ca-4f7e-8f08-0107074dd1ed" (UID: "df87ea2e-03ca-4f7e-8f08-0107074dd1ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:21.994183 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.994159 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca" pod "df87ea2e-03ca-4f7e-8f08-0107074dd1ed" (UID: "df87ea2e-03ca-4f7e-8f08-0107074dd1ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:21.994269 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:21.994254 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt" pod "df87ea2e-03ca-4f7e-8f08-0107074dd1ed" (UID: "df87ea2e-03ca-4f7e-8f08-0107074dd1ed"). InnerVolumeSpecName "kube-api-access-zmrnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:22.093089 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.093009 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:22.093089 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.093036 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:22.093089 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.093046 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrnt\" (UniqueName: \"kubernetes.io/projected/df87ea2e-03ca-4f7e-8f08-0107074dd1ed-kube-api-access-zmrnt\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:22.807248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.807214 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" Nov 03 22:03:22.807248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.807223 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm" event={"ID":"df87ea2e-03ca-4f7e-8f08-0107074dd1ed","Type":"ContainerDied","Data":"5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180"} Nov 03 22:03:22.807484 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:22.807276 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a417a831ccfd786b33c10d56e3645774cd2755d1a4ef6945a34e744a7b17180" Nov 03 22:03:29.656928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:29.656893 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:34.781556 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:34.781504 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:37.707952 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.707915 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9"] Nov 03 22:03:37.708406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.708371 5973 cpu_manager.go:414] "containerMap: removing container" podUID="df87ea2e-03ca-4f7e-8f08-0107074dd1ed" containerName="revoke-keys" Nov 03 22:03:37.708406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.708384 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df87ea2e-03ca-4f7e-8f08-0107074dd1ed" containerName="revoke-keys" Nov 03 22:03:37.708545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.708449 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="df87ea2e-03ca-4f7e-8f08-0107074dd1ed" containerName="revoke-keys" Nov 03 22:03:37.711912 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.711891 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.716749 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.716719 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:03:37.720520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.717681 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:03:37.720520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.717701 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:03:37.720520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.717740 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:03:37.723846 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.723818 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9"] Nov 03 22:03:37.767326 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.767298 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhs8\" (UniqueName: \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.767513 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.767339 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.767513 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.767494 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.868452 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.868418 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.868647 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.868515 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.868647 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.868573 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhs8\" (UniqueName: \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.869057 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.869033 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.870967 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.870944 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:37.880770 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:37.880747 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhs8\" (UniqueName: \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\") pod \"maas-api-revoke-sub-79805ca9-tsmg9\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:38.035642 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:38.035563 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:38.316302 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:38.316275 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9"] Nov 03 22:03:38.316978 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:03:38.316941 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2266f37b_4d68_4b0c_b920_b964f2a4b1a1.slice/crio-f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5 WatchSource:0}: Error finding container f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5: Status 404 returned error can't find the container with id f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5 Nov 03 22:03:38.937500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:38.937449 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" event={"ID":"2266f37b-4d68-4b0c-b920-b964f2a4b1a1","Type":"ContainerStarted","Data":"e4478c92c4c1efbd3cfde3bc431513f4949559558fb375d312959767c64afd85"} Nov 03 22:03:38.937889 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:38.937526 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" event={"ID":"2266f37b-4d68-4b0c-b920-b964f2a4b1a1","Type":"ContainerStarted","Data":"f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5"} Nov 03 22:03:38.956907 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:38.956861 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" podStartSLOduration=1.956844193 podStartE2EDuration="1.956844193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:38.954808498 +0000 UTC m=+1058.729658346" watchObservedRunningTime="2026-11-03 22:03:38.956844193 +0000 UTC m=+1058.731694040" Nov 03 22:03:39.945542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:39.945507 5973 generic.go:356] "Generic (PLEG): container finished" podID="2266f37b-4d68-4b0c-b920-b964f2a4b1a1" containerID="e4478c92c4c1efbd3cfde3bc431513f4949559558fb375d312959767c64afd85" exitCode=0 Nov 03 22:03:39.946006 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:39.945589 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" event={"ID":"2266f37b-4d68-4b0c-b920-b964f2a4b1a1","Type":"ContainerDied","Data":"e4478c92c4c1efbd3cfde3bc431513f4949559558fb375d312959767c64afd85"} Nov 03 22:03:41.082203 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.082171 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:41.199883 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.199850 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\" (UniqueName: \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\") pod \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " Nov 03 22:03:41.200085 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.199904 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\") pod \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " Nov 03 22:03:41.200085 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.199948 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\") pod \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\" (UID: \"2266f37b-4d68-4b0c-b920-b964f2a4b1a1\") " Nov 03 22:03:41.200394 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.200358 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca" pod "2266f37b-4d68-4b0c-b920-b964f2a4b1a1" (UID: "2266f37b-4d68-4b0c-b920-b964f2a4b1a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:41.202226 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.202171 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8" pod "2266f37b-4d68-4b0c-b920-b964f2a4b1a1" (UID: "2266f37b-4d68-4b0c-b920-b964f2a4b1a1"). InnerVolumeSpecName "kube-api-access-rfhs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:41.202226 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.202201 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca" pod "2266f37b-4d68-4b0c-b920-b964f2a4b1a1" (UID: "2266f37b-4d68-4b0c-b920-b964f2a4b1a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:41.300874 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.300843 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhs8\" (UniqueName: \"kubernetes.io/projected/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-kube-api-access-rfhs8\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:41.300874 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.300871 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:41.300874 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.300880 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2266f37b-4d68-4b0c-b920-b964f2a4b1a1-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:41.959368 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.959343 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" Nov 03 22:03:41.959570 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.959414 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9" event={"ID":"2266f37b-4d68-4b0c-b920-b964f2a4b1a1","Type":"ContainerDied","Data":"f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5"} Nov 03 22:03:41.959570 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:41.959479 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90e62bcd5705c72f12abac0a6dd78b51e5bed7c139ca29b56ded70531017ac5" Nov 03 22:03:44.400068 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:03:44.400030 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:03:48.449417 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:48.449383 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:52.004485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:52.004373 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:03:55.492036 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.492001 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j"] Nov 03 22:03:55.492539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.492450 5973 cpu_manager.go:414] "containerMap: removing container" podUID="2266f37b-4d68-4b0c-b920-b964f2a4b1a1" containerName="revoke-keys" Nov 03 22:03:55.492539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.492475 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2266f37b-4d68-4b0c-b920-b964f2a4b1a1" containerName="revoke-keys" Nov 03 22:03:55.492688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.492584 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="2266f37b-4d68-4b0c-b920-b964f2a4b1a1" containerName="revoke-keys" Nov 03 22:03:55.496341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.496322 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.509535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.505232 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:03:55.509535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.505318 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:03:55.509535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.505838 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:03:55.509535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.506151 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:03:55.514063 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.514030 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j"] Nov 03 22:03:55.554002 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.553964 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.554002 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.554009 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwblc\" (UniqueName: \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.554241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.554131 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.655520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.655448 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.655721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.655532 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwblc\" (UniqueName: \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.655721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.655573 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.656168 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.656141 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.658012 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.657987 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.667959 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.667938 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwblc\" (UniqueName: \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\") pod \"maas-api-revoke-sub-0b7e8edb-phg5j\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:55.826502 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:55.826376 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:56.104991 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:56.104961 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j"] Nov 03 22:03:56.105409 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:03:56.105380 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969200da_2b26_4dd9_98af_9b4603a8b349.slice/crio-b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff WatchSource:0}: Error finding container b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff: Status 404 returned error can't find the container with id b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff Nov 03 22:03:56.654824 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:56.654796 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:57.069810 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:57.069728 5973 generic.go:356] "Generic (PLEG): container finished" podID="969200da-2b26-4dd9-98af-9b4603a8b349" containerID="b1323584cc5db8044e5b58839057be7ed99a8eea052862454cdf7574cb17221f" exitCode=0 Nov 03 22:03:57.069810 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:57.069796 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" event={"ID":"969200da-2b26-4dd9-98af-9b4603a8b349","Type":"ContainerDied","Data":"b1323584cc5db8044e5b58839057be7ed99a8eea052862454cdf7574cb17221f"} Nov 03 22:03:57.070006 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:57.069858 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" event={"ID":"969200da-2b26-4dd9-98af-9b4603a8b349","Type":"ContainerStarted","Data":"b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff"} Nov 03 22:03:58.215171 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.215142 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:58.279810 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.279765 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\") pod \"969200da-2b26-4dd9-98af-9b4603a8b349\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " Nov 03 22:03:58.279986 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.279845 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\") pod \"969200da-2b26-4dd9-98af-9b4603a8b349\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " Nov 03 22:03:58.279986 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.279881 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\" (UniqueName: \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\") pod \"969200da-2b26-4dd9-98af-9b4603a8b349\" (UID: \"969200da-2b26-4dd9-98af-9b4603a8b349\") " Nov 03 22:03:58.280223 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.280194 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca" pod "969200da-2b26-4dd9-98af-9b4603a8b349" (UID: "969200da-2b26-4dd9-98af-9b4603a8b349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:58.282183 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.282160 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca" pod "969200da-2b26-4dd9-98af-9b4603a8b349" (UID: "969200da-2b26-4dd9-98af-9b4603a8b349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:58.282278 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.282188 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc" pod "969200da-2b26-4dd9-98af-9b4603a8b349" (UID: "969200da-2b26-4dd9-98af-9b4603a8b349"). InnerVolumeSpecName "kube-api-access-xwblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:58.380826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.380785 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:58.380826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.380821 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969200da-2b26-4dd9-98af-9b4603a8b349-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:58.380826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:58.380831 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwblc\" (UniqueName: \"kubernetes.io/projected/969200da-2b26-4dd9-98af-9b4603a8b349-kube-api-access-xwblc\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.084664 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:59.084631 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" Nov 03 22:03:59.084664 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:59.084659 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j" event={"ID":"969200da-2b26-4dd9-98af-9b4603a8b349","Type":"ContainerDied","Data":"b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff"} Nov 03 22:03:59.084896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:03:59.084706 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e382b299cac5b7e80d2a83dbd79a017c6b7e428c24f064e26602d6bae235ff" Nov 03 22:04:01.018659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:01.018627 5973 scope.go:122] "RemoveContainer" containerID="af74429d511493ed6f7a61d9e9504f7b07e649e96b475f37a7b9a0674eeb9f91" Nov 03 22:04:03.036099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.036062 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns"] Nov 03 22:04:03.041385 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.041357 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct"] Nov 03 22:04:03.044640 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.044615 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j"] Nov 03 22:04:03.047165 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.047139 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51dd77e1-qkxns"] Nov 03 22:04:03.049573 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.049552 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2f659d-vlnct"] Nov 03 22:04:03.051900 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:03.051880 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5cf1b-jfl6j"] Nov 03 22:04:04.345342 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:04.345307 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128f001f-d922-4965-a4af-d30c8390962a" path="/var/lib/kubelet/pods/128f001f-d922-4965-a4af-d30c8390962a/volumes" Nov 03 22:04:04.345766 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:04.345708 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6341c596-5a6e-48c2-ba39-fd728b9e7b3e" path="/var/lib/kubelet/pods/6341c596-5a6e-48c2-ba39-fd728b9e7b3e/volumes" Nov 03 22:04:04.346054 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:04.346041 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4ee6ae-3404-48f5-9f8a-04d3e33286ae" path="/var/lib/kubelet/pods/8c4ee6ae-3404-48f5-9f8a-04d3e33286ae/volumes" Nov 03 22:04:06.175171 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.175135 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh"] Nov 03 22:04:06.175608 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.175564 5973 cpu_manager.go:414] "containerMap: removing container" podUID="969200da-2b26-4dd9-98af-9b4603a8b349" containerName="revoke-keys" Nov 03 22:04:06.175608 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.175577 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969200da-2b26-4dd9-98af-9b4603a8b349" containerName="revoke-keys" Nov 03 22:04:06.175676 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.175651 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="969200da-2b26-4dd9-98af-9b4603a8b349" containerName="revoke-keys" Nov 03 22:04:06.180257 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.180234 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.185280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.185256 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:04:06.185445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.185296 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:04:06.185559 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.185308 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:04:06.185559 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.185339 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:04:06.189157 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.189136 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh"] Nov 03 22:04:06.249228 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.249196 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:06.263131 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.263104 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.263280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.263138 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.263280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.263172 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqp8f\" (UniqueName: \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.364305 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.364277 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqp8f\" (UniqueName: \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.364657 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.364618 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.364781 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.364662 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.365206 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.365186 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.367148 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.367129 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.376649 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.376630 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqp8f\" (UniqueName: \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\") pod \"maas-api-revoke-sub-3d4156f1-tw8vh\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.495846 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.495759 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:06.776881 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:06.776794 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh"] Nov 03 22:04:06.782752 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:04:06.782711 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09dc10b_f345_4034_8e7c_ad6c35b454ee.slice/crio-9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f WatchSource:0}: Error finding container 9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f: Status 404 returned error can't find the container with id 9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f Nov 03 22:04:07.137601 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:07.137561 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" event={"ID":"a09dc10b-f345-4034-8e7c-ad6c35b454ee","Type":"ContainerStarted","Data":"e01449a47b21b4afdbd2a8880bebcada3805b34f2804d06f81bd0bc5ae7e1c34"} Nov 03 22:04:07.137762 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:07.137621 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" event={"ID":"a09dc10b-f345-4034-8e7c-ad6c35b454ee","Type":"ContainerStarted","Data":"9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f"} Nov 03 22:04:07.159796 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:07.159748 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" podStartSLOduration=1.159734184 podStartE2EDuration="1.159734184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:07.15717302 +0000 UTC m=+1086.932022868" watchObservedRunningTime="2026-11-03 22:04:07.159734184 +0000 UTC m=+1086.934584071" Nov 03 22:04:08.145771 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:08.145737 5973 generic.go:356] "Generic (PLEG): container finished" podID="a09dc10b-f345-4034-8e7c-ad6c35b454ee" containerID="e01449a47b21b4afdbd2a8880bebcada3805b34f2804d06f81bd0bc5ae7e1c34" exitCode=0 Nov 03 22:04:08.146185 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:08.145825 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" event={"ID":"a09dc10b-f345-4034-8e7c-ad6c35b454ee","Type":"ContainerDied","Data":"e01449a47b21b4afdbd2a8880bebcada3805b34f2804d06f81bd0bc5ae7e1c34"} Nov 03 22:04:09.285626 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.285602 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:09.394976 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.394943 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\") pod \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " Nov 03 22:04:09.395147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.394998 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\" (UniqueName: \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\") pod \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " Nov 03 22:04:09.395147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.395036 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\") pod \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\" (UID: \"a09dc10b-f345-4034-8e7c-ad6c35b454ee\") " Nov 03 22:04:09.395339 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.395316 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca" pod "a09dc10b-f345-4034-8e7c-ad6c35b454ee" (UID: "a09dc10b-f345-4034-8e7c-ad6c35b454ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:09.397378 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.397342 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca" pod "a09dc10b-f345-4034-8e7c-ad6c35b454ee" (UID: "a09dc10b-f345-4034-8e7c-ad6c35b454ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:09.397378 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.397359 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f" pod "a09dc10b-f345-4034-8e7c-ad6c35b454ee" (UID: "a09dc10b-f345-4034-8e7c-ad6c35b454ee"). InnerVolumeSpecName "kube-api-access-pqp8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:09.496751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.496717 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:09.496751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.496746 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqp8f\" (UniqueName: \"kubernetes.io/projected/a09dc10b-f345-4034-8e7c-ad6c35b454ee-kube-api-access-pqp8f\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:09.496751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:09.496756 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09dc10b-f345-4034-8e7c-ad6c35b454ee-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:10.160308 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:10.160282 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" Nov 03 22:04:10.160503 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:10.160279 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh" event={"ID":"a09dc10b-f345-4034-8e7c-ad6c35b454ee","Type":"ContainerDied","Data":"9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f"} Nov 03 22:04:10.160503 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:10.160414 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b2baff1c527a7e9a85dfc843a523310043e8ed8ee7dc1c27d517dfe3c89ad2f" Nov 03 22:04:13.023249 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:13.023210 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828"] Nov 03 22:04:13.026281 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:13.026256 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b31400-lf828"] Nov 03 22:04:14.345070 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:14.345036 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74433e7-c72b-46e2-9b2f-a64eb8c052ed" path="/var/lib/kubelet/pods/f74433e7-c72b-46e2-9b2f-a64eb8c052ed/volumes" Nov 03 22:04:14.398636 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:04:14.398602 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:04:15.024016 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:15.023976 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6"] Nov 03 22:04:15.026689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:15.026671 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872718cf-4v5c6"] Nov 03 22:04:16.344212 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:16.344180 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5007aadd-1499-42d9-9d85-0f7f4c42ce32" path="/var/lib/kubelet/pods/5007aadd-1499-42d9-9d85-0f7f4c42ce32/volumes" Nov 03 22:04:16.934924 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:16.934896 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:20.025019 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:20.024977 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz"] Nov 03 22:04:20.029966 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:20.029929 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d76ccb-xm6dz"] Nov 03 22:04:20.344823 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:20.344747 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de99fca-5d51-4b4b-beb5-be19b67ece7f" path="/var/lib/kubelet/pods/9de99fca-5d51-4b4b-beb5-be19b67ece7f/volumes" Nov 03 22:04:21.026857 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:21.026821 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8"] Nov 03 22:04:21.030945 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:21.030923 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd73d6f-hq4h8"] Nov 03 22:04:21.358722 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:21.358640 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:22.345076 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:22.345040 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98bc69c-1cdc-493b-aab7-df5ecd79fc37" path="/var/lib/kubelet/pods/d98bc69c-1cdc-493b-aab7-df5ecd79fc37/volumes" Nov 03 22:04:22.965797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:22.965760 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:24.214279 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.214240 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh"] Nov 03 22:04:24.219358 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.219332 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tw8vh"] Nov 03 22:04:24.238928 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.238904 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn"] Nov 03 22:04:24.239296 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.239283 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a09dc10b-f345-4034-8e7c-ad6c35b454ee" containerName="revoke-keys" Nov 03 22:04:24.239338 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.239297 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09dc10b-f345-4034-8e7c-ad6c35b454ee" containerName="revoke-keys" Nov 03 22:04:24.239384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.239356 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09dc10b-f345-4034-8e7c-ad6c35b454ee" containerName="revoke-keys" Nov 03 22:04:24.243728 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.243712 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.248653 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.248629 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:04:24.249970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.249605 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:04:24.249970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.249652 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:04:24.249970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.249788 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:04:24.252580 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.252559 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn"] Nov 03 22:04:24.344793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.344757 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09dc10b-f345-4034-8e7c-ad6c35b454ee" path="/var/lib/kubelet/pods/a09dc10b-f345-4034-8e7c-ad6c35b454ee/volumes" Nov 03 22:04:24.367128 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.367105 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dq5x\" (UniqueName: \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.367227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.367159 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.367266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.367225 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.468069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.467977 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dq5x\" (UniqueName: \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.468069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.468058 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.468290 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.468087 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.468725 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.468698 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.470529 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.470509 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.480535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.480515 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dq5x\" (UniqueName: \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\") pod \"maas-api-revoke-sub-3d4156f1-4wctn\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.558755 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.558723 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:24.835731 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:24.835700 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn"] Nov 03 22:04:24.837080 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:04:24.837055 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2682cab_dfd5_43d5_bb3c_7256fbff3a7f.slice/crio-4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec WatchSource:0}: Error finding container 4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec: Status 404 returned error can't find the container with id 4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec Nov 03 22:04:25.270376 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:25.270338 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" event={"ID":"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f","Type":"ContainerStarted","Data":"b34b1f3618c7a9f84c7708b5e45b838d01b4ab3489ad1ee06cfe82516a3d0f5e"} Nov 03 22:04:25.270797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:25.270400 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" event={"ID":"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f","Type":"ContainerStarted","Data":"4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec"} Nov 03 22:04:25.289475 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:25.289403 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" podStartSLOduration=1.289389866 podStartE2EDuration="1.289389866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:25.288097453 +0000 UTC m=+1105.062947301" watchObservedRunningTime="2026-11-03 22:04:25.289389866 +0000 UTC m=+1105.064239761" Nov 03 22:04:26.278268 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:26.278234 5973 generic.go:356] "Generic (PLEG): container finished" podID="d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" containerID="b34b1f3618c7a9f84c7708b5e45b838d01b4ab3489ad1ee06cfe82516a3d0f5e" exitCode=0 Nov 03 22:04:26.278698 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:26.278326 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" event={"ID":"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f","Type":"ContainerDied","Data":"b34b1f3618c7a9f84c7708b5e45b838d01b4ab3489ad1ee06cfe82516a3d0f5e"} Nov 03 22:04:27.419208 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.419185 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:27.498105 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.498075 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\") pod \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " Nov 03 22:04:27.498297 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.498156 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\") pod \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " Nov 03 22:04:27.498297 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.498182 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\" (UniqueName: \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\") pod \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\" (UID: \"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f\") " Nov 03 22:04:27.498470 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.498424 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca" pod "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" (UID: "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:27.500425 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.500393 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca" pod "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" (UID: "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:27.500570 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.500547 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x" pod "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" (UID: "d2682cab-dfd5-43d5-bb3c-7256fbff3a7f"). InnerVolumeSpecName "kube-api-access-9dq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:27.599580 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.599507 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:27.599580 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.599534 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:27.599580 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:27.599544 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dq5x\" (UniqueName: \"kubernetes.io/projected/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f-kube-api-access-9dq5x\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:28.291906 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:28.291881 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" Nov 03 22:04:28.292096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:28.291925 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn" event={"ID":"d2682cab-dfd5-43d5-bb3c-7256fbff3a7f","Type":"ContainerDied","Data":"4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec"} Nov 03 22:04:28.292096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:28.291978 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5adff4585b0e7dd5de5538ba770253071c17afb6b81ae802e56c2f946409ec" Nov 03 22:04:33.032143 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:33.032101 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq"] Nov 03 22:04:33.035011 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:33.034987 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659e34db-nxxjq"] Nov 03 22:04:34.024536 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:34.024499 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d"] Nov 03 22:04:34.027488 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:34.027451 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3913c2-dn64d"] Nov 03 22:04:34.346152 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:34.346077 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db71e8ed-c217-4c3d-a337-01e5f76a991c" path="/var/lib/kubelet/pods/db71e8ed-c217-4c3d-a337-01e5f76a991c/volumes" Nov 03 22:04:34.346544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:34.346492 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0e0b23-f5be-4b1f-988b-d6c0045c10b5" path="/var/lib/kubelet/pods/fe0e0b23-f5be-4b1f-988b-d6c0045c10b5/volumes" Nov 03 22:04:34.947605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:34.947571 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:36.023986 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.023949 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv"] Nov 03 22:04:36.026694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.026663 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d902e139-wftrv"] Nov 03 22:04:36.028986 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.028959 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9"] Nov 03 22:04:36.031044 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.031027 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4aeb0e-rdqr9"] Nov 03 22:04:36.344683 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.344599 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3224cbfb-42f0-45bd-a293-becc7d2cb02e" path="/var/lib/kubelet/pods/3224cbfb-42f0-45bd-a293-becc7d2cb02e/volumes" Nov 03 22:04:36.345213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:36.345191 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568ba166-ca39-459f-94d8-b0ded051adc9" path="/var/lib/kubelet/pods/568ba166-ca39-459f-94d8-b0ded051adc9/volumes" Nov 03 22:04:37.028515 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:37.028479 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d"] Nov 03 22:04:37.031756 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:37.031732 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82911aa3-z2f4d"] Nov 03 22:04:37.955774 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:37.955746 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:38.344956 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:38.344884 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5185f19-313f-414f-b642-c9525ae8adfa" path="/var/lib/kubelet/pods/e5185f19-313f-414f-b642-c9525ae8adfa/volumes" Nov 03 22:04:39.026681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:39.026641 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69"] Nov 03 22:04:39.028886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:39.028865 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe429fe1-p5r69"] Nov 03 22:04:40.030746 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.030710 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4"] Nov 03 22:04:40.033241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.033217 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113d05e7-bq4s4"] Nov 03 22:04:40.035558 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.035532 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949"] Nov 03 22:04:40.037484 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.037446 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e1a97ac-jp949"] Nov 03 22:04:40.345228 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.345138 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4922b904-9dbc-4b27-bb51-741ef7cbae57" path="/var/lib/kubelet/pods/4922b904-9dbc-4b27-bb51-741ef7cbae57/volumes" Nov 03 22:04:40.345645 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.345625 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6906641a-8aa0-43f1-a26d-75cef444a128" path="/var/lib/kubelet/pods/6906641a-8aa0-43f1-a26d-75cef444a128/volumes" Nov 03 22:04:40.346061 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:40.346043 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e91c153-32da-4427-9ea7-633dba8c415b" path="/var/lib/kubelet/pods/8e91c153-32da-4427-9ea7-633dba8c415b/volumes" Nov 03 22:04:44.023921 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.023885 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5"] Nov 03 22:04:44.026576 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.026554 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d351fb-4hrh5"] Nov 03 22:04:44.344447 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.344370 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299392bc-7989-43ed-93cb-24829eb17e23" path="/var/lib/kubelet/pods/299392bc-7989-43ed-93cb-24829eb17e23/volumes" Nov 03 22:04:44.400114 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:04:44.400071 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:04:44.721763 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.721725 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn"] Nov 03 22:04:44.726710 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.726685 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4wctn"] Nov 03 22:04:44.743445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.743419 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm"] Nov 03 22:04:44.743975 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.743952 5973 cpu_manager.go:414] "containerMap: removing container" podUID="d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" containerName="revoke-keys" Nov 03 22:04:44.743975 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.743974 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" containerName="revoke-keys" Nov 03 22:04:44.744131 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.744051 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" containerName="revoke-keys" Nov 03 22:04:44.749217 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.749199 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.756818 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.756481 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:04:44.757265 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.757245 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:04:44.760479 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.760432 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm"] Nov 03 22:04:44.766761 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.766743 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:04:44.766869 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.766800 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:04:44.865628 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.865588 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.865801 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.865635 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jf9h\" (UniqueName: \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.865801 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.865667 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.966474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.966422 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.966653 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.966485 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jf9h\" (UniqueName: \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.966653 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.966519 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.967219 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.967200 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.969013 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.968994 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:44.979215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:44.979158 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jf9h\" (UniqueName: \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\") pod \"maas-api-revoke-sub-3d4156f1-wvghm\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:45.064868 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:45.064836 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:45.346641 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:45.346617 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm"] Nov 03 22:04:45.348045 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:04:45.348019 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f374b9_05c0_4eb1_a8a5_15ada787e425.slice/crio-d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3 WatchSource:0}: Error finding container d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3: Status 404 returned error can't find the container with id d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3 Nov 03 22:04:45.404191 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:45.404157 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" event={"ID":"a8f374b9-05c0-4eb1-a8a5-15ada787e425","Type":"ContainerStarted","Data":"d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3"} Nov 03 22:04:46.344118 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:46.344075 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2682cab-dfd5-43d5-bb3c-7256fbff3a7f" path="/var/lib/kubelet/pods/d2682cab-dfd5-43d5-bb3c-7256fbff3a7f/volumes" Nov 03 22:04:46.411757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:46.411724 5973 generic.go:356] "Generic (PLEG): container finished" podID="a8f374b9-05c0-4eb1-a8a5-15ada787e425" containerID="6638a8ac7afef00ddc111cbb74bdd93bc91a801a7294d6123ac87d3c2e988aff" exitCode=0 Nov 03 22:04:46.411955 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:46.411807 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" event={"ID":"a8f374b9-05c0-4eb1-a8a5-15ada787e425","Type":"ContainerDied","Data":"6638a8ac7afef00ddc111cbb74bdd93bc91a801a7294d6123ac87d3c2e988aff"} Nov 03 22:04:47.552308 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.552286 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:47.695103 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.695004 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\") pod \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " Nov 03 22:04:47.695103 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.695084 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\" (UniqueName: \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\") pod \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " Nov 03 22:04:47.695103 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.695106 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\") pod \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\" (UID: \"a8f374b9-05c0-4eb1-a8a5-15ada787e425\") " Nov 03 22:04:47.695560 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.695528 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca" pod "a8f374b9-05c0-4eb1-a8a5-15ada787e425" (UID: "a8f374b9-05c0-4eb1-a8a5-15ada787e425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:47.697427 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.697401 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca" pod "a8f374b9-05c0-4eb1-a8a5-15ada787e425" (UID: "a8f374b9-05c0-4eb1-a8a5-15ada787e425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:47.697545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.697430 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h" pod "a8f374b9-05c0-4eb1-a8a5-15ada787e425" (UID: "a8f374b9-05c0-4eb1-a8a5-15ada787e425"). InnerVolumeSpecName "kube-api-access-7jf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:47.796488 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.796432 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:47.796651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.796483 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jf9h\" (UniqueName: \"kubernetes.io/projected/a8f374b9-05c0-4eb1-a8a5-15ada787e425-kube-api-access-7jf9h\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:47.796651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:47.796508 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f374b9-05c0-4eb1-a8a5-15ada787e425-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:48.425552 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:48.425520 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" Nov 03 22:04:48.425707 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:48.425520 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm" event={"ID":"a8f374b9-05c0-4eb1-a8a5-15ada787e425","Type":"ContainerDied","Data":"d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3"} Nov 03 22:04:48.425707 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:48.425654 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43d87060f69a3ed0f439943e8c9a64fbf8e6d74bad7ef0e6a41e7c472d183e3" Nov 03 22:04:55.526242 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:55.526205 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:56.025177 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:56.025138 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw"] Nov 03 22:04:56.028667 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:56.028634 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bac426-x4svw"] Nov 03 22:04:56.344213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:56.344135 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba4ca7f-bbd2-4280-8598-ef50d031b42e" path="/var/lib/kubelet/pods/fba4ca7f-bbd2-4280-8598-ef50d031b42e/volumes" Nov 03 22:04:56.764522 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:56.764487 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:57.059294 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:57.059200 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:57.349761 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:57.349673 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:57.652845 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:57.652814 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:57.970341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:57.970246 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:58.344924 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:58.344840 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:58.729981 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:58.729906 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:04:58.810400 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:58.810363 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:59.409791 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:04:59.409761 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:00.330768 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:00.330730 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:01.047407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:01.047369 5973 scope.go:122] "RemoveContainer" containerID="2771845fb1ad6f32ae388c54fab657fec1933ae86207622d028e27ef9cfbb111" Nov 03 22:05:01.057733 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:01.057715 5973 scope.go:122] "RemoveContainer" containerID="af953e630a774233a4eb307091990f6c32133dd28b633fbca09a6573a4bcad21" Nov 03 22:05:01.066803 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:01.066777 5973 scope.go:122] "RemoveContainer" containerID="e4766883484409825a57030c722668cb52b035dfe97f2f6ffe09ca1b6eaf5421" Nov 03 22:05:01.886581 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:01.886546 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:02.950072 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:02.950040 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.356777 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.356703 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:05:03.357144 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.357128 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a8f374b9-05c0-4eb1-a8a5-15ada787e425" containerName="revoke-keys" Nov 03 22:05:03.357194 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.357146 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f374b9-05c0-4eb1-a8a5-15ada787e425" containerName="revoke-keys" Nov 03 22:05:03.357234 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.357204 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f374b9-05c0-4eb1-a8a5-15ada787e425" containerName="revoke-keys" Nov 03 22:05:03.362016 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.361989 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.372899 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.372874 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-94e879-openshift-default-dockercfg-pzpkc\"" Nov 03 22:05:03.377024 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.376998 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:05:03.418088 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.418061 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.442394 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442365 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdj7\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442398 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442420 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442436 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442490 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442759 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442609 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442759 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442653 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442759 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442741 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.442860 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.442762 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544001 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.543964 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdj7\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544001 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544004 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544270 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544024 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544270 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544044 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544270 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544193 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544270 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544251 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544290 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544385 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544419 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544507 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544706 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544685 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544798 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544780 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544798 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544788 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.544896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.544862 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.546653 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.546628 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.547030 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.547007 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.562314 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.562288 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.562578 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.562556 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdj7\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\") pod \"e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.675286 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.675256 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:03.685595 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.685563 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.959296 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.959267 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:05:03.960904 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:03.960878 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb0ef56_489f_454a_8cf5_29fffaaf7671.slice/crio-2dff716d2f7fdeccb3b6c26c3bd2d32f00e42b4111116ef864a105ad0bd534d8 WatchSource:0}: Error finding container 2dff716d2f7fdeccb3b6c26c3bd2d32f00e42b4111116ef864a105ad0bd534d8: Status 404 returned error can't find the container with id 2dff716d2f7fdeccb3b6c26c3bd2d32f00e42b4111116ef864a105ad0bd534d8 Nov 03 22:05:03.963046 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.963015 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:03.963148 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.963093 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:03.963148 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:03.963138 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:04.528195 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:04.528157 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerStarted","Data":"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4"} Nov 03 22:05:04.528377 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:04.528213 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerStarted","Data":"2dff716d2f7fdeccb3b6c26c3bd2d32f00e42b4111116ef864a105ad0bd534d8"} Nov 03 22:05:04.557948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:04.557901 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podStartSLOduration=1.557886761 podStartE2EDuration="1.557886761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:04.555808117 +0000 UTC m=+1144.330658003" watchObservedRunningTime="2026-11-03 22:05:04.557886761 +0000 UTC m=+1144.332736608" Nov 03 22:05:04.675475 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:04.675430 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:04.725250 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:04.725196 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:05.029820 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:05.029778 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc"] Nov 03 22:05:05.032229 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:05.032208 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c26b79c-r6xrc"] Nov 03 22:05:05.680395 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:05.680367 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:06.345072 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:06.345043 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390589f8-71d7-451e-8728-13447c6be485" path="/var/lib/kubelet/pods/390589f8-71d7-451e-8728-13447c6be485/volumes" Nov 03 22:05:06.540723 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:06.540695 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:06.541783 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:06.541765 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:06.881831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:06.881795 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:07.216863 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:07.216827 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.099992 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.099958 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.162862 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.162827 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:05:08.166829 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.166810 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.174105 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.174081 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-94e879\"" Nov 03 22:05:08.174252 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.174175 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-223faddd-dockercfg-n5zwd\"" Nov 03 22:05:08.177500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.177478 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:05:08.189610 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.189575 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.190073 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.189655 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjnm\" (UniqueName: \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.207087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.207056 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:05:08.210705 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.210685 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.223798 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.223775 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:05:08.290660 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.290625 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2rj\" (UniqueName: \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.290819 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.290671 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.290819 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.290700 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.290911 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.290817 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjnm\" (UniqueName: \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.291351 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.291330 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.304127 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.304095 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjnm\" (UniqueName: \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\") pod \"payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.392145 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.392115 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2rj\" (UniqueName: \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.392330 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.392166 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.393174 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.393154 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.404971 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.404937 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2rj\" (UniqueName: \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\") pod \"payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.406432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.406403 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.481520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.481489 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:08.528734 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.528698 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:08.766806 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.766779 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:05:08.767735 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:08.767709 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3278887b_3642_4447_8686_b571ca2eebe6.slice/crio-912bf974ec4fb12f93c9a3dcd42ed75119bbd10e1c2b4a8ec146099742e7be56 WatchSource:0}: Error finding container 912bf974ec4fb12f93c9a3dcd42ed75119bbd10e1c2b4a8ec146099742e7be56: Status 404 returned error can't find the container with id 912bf974ec4fb12f93c9a3dcd42ed75119bbd10e1c2b4a8ec146099742e7be56 Nov 03 22:05:08.822703 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:08.822663 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:05:08.828727 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:08.828699 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2951cf_e822_449b_948b_4f1c858ebd7c.slice/crio-b8d9baa4beba0e8a957fcb520e8ab56078af092d4f78798619c2f07db6b8b7cf WatchSource:0}: Error finding container b8d9baa4beba0e8a957fcb520e8ab56078af092d4f78798619c2f07db6b8b7cf: Status 404 returned error can't find the container with id b8d9baa4beba0e8a957fcb520e8ab56078af092d4f78798619c2f07db6b8b7cf Nov 03 22:05:09.016301 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.016215 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:09.571521 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.571486 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" event={"ID":"7c2951cf-e822-449b-948b-4f1c858ebd7c","Type":"ContainerStarted","Data":"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8"} Nov 03 22:05:09.571963 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.571551 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" event={"ID":"7c2951cf-e822-449b-948b-4f1c858ebd7c","Type":"ContainerStarted","Data":"b8d9baa4beba0e8a957fcb520e8ab56078af092d4f78798619c2f07db6b8b7cf"} Nov 03 22:05:09.571963 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.571622 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:09.573110 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.573084 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" event={"ID":"3278887b-3642-4447-8686-b571ca2eebe6","Type":"ContainerStarted","Data":"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a"} Nov 03 22:05:09.573235 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.573124 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" event={"ID":"3278887b-3642-4447-8686-b571ca2eebe6","Type":"ContainerStarted","Data":"912bf974ec4fb12f93c9a3dcd42ed75119bbd10e1c2b4a8ec146099742e7be56"} Nov 03 22:05:09.573235 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.573165 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:09.593315 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.593267 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" podStartSLOduration=1.593254322 podStartE2EDuration="1.593254322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:09.589661662 +0000 UTC m=+1149.364511509" watchObservedRunningTime="2026-11-03 22:05:09.593254322 +0000 UTC m=+1149.368104179" Nov 03 22:05:09.609684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.609639 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" podStartSLOduration=1.6096298070000001 podStartE2EDuration="1.609629807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:09.607385387 +0000 UTC m=+1149.382235261" watchObservedRunningTime="2026-11-03 22:05:09.609629807 +0000 UTC m=+1149.384479654" Nov 03 22:05:09.854625 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:09.854546 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.669525 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:10.669493 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.841064 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:10.841025 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:05:11.372950 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:11.372916 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:12.182153 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:12.182116 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:12.949316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:12.949280 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:13.595858 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:13.595826 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:14.397300 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:14.397253 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:14.407081 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:05:14.407047 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:05:15.079816 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:15.079780 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:15.881850 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:15.881806 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:16.676015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:16.675980 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:17.415186 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:17.415156 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:18.170542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:18.170508 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:18.877103 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:18.877062 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:19.532522 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:19.532483 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:20.387131 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.387095 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:20.582651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.582574 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:05:20.582651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.582637 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:05:20.648927 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.648891 5973 generic.go:356] "Generic (PLEG): container finished" podID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerID="d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4" exitCode=0 Nov 03 22:05:20.649086 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.648933 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerDied","Data":"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4"} Nov 03 22:05:20.649325 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.649312 5973 scope.go:122] "RemoveContainer" containerID="d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4" Nov 03 22:05:20.650571 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.650546 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:20.650646 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.650608 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:20.650681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:20.650647 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:05:21.125309 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:21.125196 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:21.540864 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:21.540830 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:21.682506 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:21.682443 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerStarted","Data":"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013"} Nov 03 22:05:21.844571 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:21.844450 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:22.142899 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.142869 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:22.452612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.452528 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:22.688791 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.688761 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:22.691348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.691314 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:05:22.691548 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.691388 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 22:05:22.750926 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:22.750842 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:23.401334 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.401295 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:23.570485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.558062 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:05:23.591428 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.591388 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:05:23.591648 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.591395 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.596646 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.596613 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-94e879\"/\"kube-root-ca.crt\"" Nov 03 22:05:23.596866 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.596727 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-94e879\"/\"default-dockercfg-47lxd\"" Nov 03 22:05:23.596866 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.596795 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-94e879\"/\"openshift-service-ca.crt\"" Nov 03 22:05:23.596866 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.596839 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-94e879\"/\"simulator-single-main-94e879-kserve-self-signed-certs\"" Nov 03 22:05:23.678879 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.678781 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:05:23.679111 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.679082 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 22:05:23.692102 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692068 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.692542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692175 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.692542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692241 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.692542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692273 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkw7\" (UniqueName: \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.692542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692334 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.692542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.692472 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.732133 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.732098 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:23.793148 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793110 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793333 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793173 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkw7\" (UniqueName: \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793333 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793209 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793333 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793250 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793493 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793345 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793493 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793393 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793858 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793828 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.793988 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.793965 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.794060 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.794002 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.796433 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.796405 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.798475 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.798436 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.810862 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.810827 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkw7\" (UniqueName: \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\") pod \"simulator-single-main-94e879-kserve-d57878c54-mgpfw\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:23.904867 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:23.904838 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:24.133423 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.133358 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:24.220430 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.220383 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:05:24.237645 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:24.237420 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64a4fdf8_0ea5_4f89_a9c6_687f52487a2a.slice/crio-93d47398bc84b06435da9489d38af70ee34c63bb9588b8e295ba63629ecefb4e WatchSource:0}: Error finding container 93d47398bc84b06435da9489d38af70ee34c63bb9588b8e295ba63629ecefb4e: Status 404 returned error can't find the container with id 93d47398bc84b06435da9489d38af70ee34c63bb9588b8e295ba63629ecefb4e Nov 03 22:05:24.681044 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.680950 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:05:24.681044 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.681032 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 22:05:24.737575 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.737536 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" event={"ID":"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a","Type":"ContainerStarted","Data":"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e"} Nov 03 22:05:24.737972 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.737588 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" event={"ID":"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a","Type":"ContainerStarted","Data":"93d47398bc84b06435da9489d38af70ee34c63bb9588b8e295ba63629ecefb4e"} Nov 03 22:05:24.774485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.765147 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" podStartSLOduration=1.765128035 podStartE2EDuration="1.765128035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:24.763791695 +0000 UTC m=+1164.538641554" watchObservedRunningTime="2026-11-03 22:05:24.765128035 +0000 UTC m=+1164.539977872" Nov 03 22:05:24.901319 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.901287 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:24.905542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:24.905522 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:25.484924 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:25.484892 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:05:25.680487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:25.677923 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:05:25.680487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:25.677987 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 22:05:25.727591 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:25.727563 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:26.625622 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:26.625582 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:26.680432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:26.679953 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:05:26.680432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:26.680016 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 22:05:27.506241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:27.506202 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:27.679666 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:27.679640 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:27.680047 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:27.679992 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:27.680740 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:27.680722 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:05:28.181151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:28.181107 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:28.974336 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:28.974300 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:29.728112 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:29.728071 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:30.390028 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:30.389975 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:31.187044 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:31.187011 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:31.839214 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:31.839179 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:32.584710 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:32.584673 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:33.311382 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:33.311354 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:33.905508 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:33.905476 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:33.918426 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:33.918397 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:33.969637 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:33.969605 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:34.837080 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:34.837045 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:05:34.888923 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:34.888890 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:35.565683 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:35.565644 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:36.211718 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:36.211681 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:36.874620 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:36.874591 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:37.544914 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:37.544877 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:38.203951 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:38.203916 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:38.868638 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:38.868606 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:39.671358 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:39.671324 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:40.431144 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:40.431068 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:40.859806 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:40.859727 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:05:41.112087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:41.111999 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:41.775077 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:41.775045 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:42.612797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:42.612759 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:43.413695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:43.413659 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:44.172477 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.171436 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:44.222392 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.222359 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g"] Nov 03 22:05:44.226020 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.225997 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.229916 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.229890 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:05:44.230023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.229948 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:05:44.232553 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.232529 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:05:44.236472 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.236424 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g"] Nov 03 22:05:44.302138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.302105 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.302305 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.302229 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7zp4\" (UniqueName: \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.302347 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.302311 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.330029 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.330001 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:05:44.399574 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:05:44.399531 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:05:44.403063 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.403041 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7zp4\" (UniqueName: \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.403163 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.403146 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.403211 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.403195 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.403816 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.403797 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.405651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.405626 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.416765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.416738 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7zp4\" (UniqueName: \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\") pod \"maas-api-revoke-sub-3900c8ee-hnv9g\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.541915 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.541843 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:44.818993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.818967 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g"] Nov 03 22:05:44.820663 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:44.820627 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e3eebe_70b1_4957_913a_ea117211d3a9.slice/crio-ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1 WatchSource:0}: Error finding container ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1: Status 404 returned error can't find the container with id ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1 Nov 03 22:05:44.820823 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.820656 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:44.897430 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:44.897390 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" event={"ID":"95e3eebe-70b1-4957-913a-ea117211d3a9","Type":"ContainerStarted","Data":"ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1"} Nov 03 22:05:45.473188 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:45.473150 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:45.905043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:45.905011 5973 generic.go:356] "Generic (PLEG): container finished" podID="95e3eebe-70b1-4957-913a-ea117211d3a9" containerID="6c543183d6ce0cdd6cf38d36069ad180ab07971f66e28771873d439173fcfe6a" exitCode=0 Nov 03 22:05:45.905419 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:45.905099 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" event={"ID":"95e3eebe-70b1-4957-913a-ea117211d3a9","Type":"ContainerDied","Data":"6c543183d6ce0cdd6cf38d36069ad180ab07971f66e28771873d439173fcfe6a"} Nov 03 22:05:45.985560 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:45.985525 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:46.295128 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:46.295034 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:46.941438 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:46.941322 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:47.052366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.052331 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:47.132657 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.132617 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\") pod \"95e3eebe-70b1-4957-913a-ea117211d3a9\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " Nov 03 22:05:47.132863 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.132671 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\" (UniqueName: \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\") pod \"95e3eebe-70b1-4957-913a-ea117211d3a9\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " Nov 03 22:05:47.132863 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.132750 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\") pod \"95e3eebe-70b1-4957-913a-ea117211d3a9\" (UID: \"95e3eebe-70b1-4957-913a-ea117211d3a9\") " Nov 03 22:05:47.133150 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.133122 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca" pod "95e3eebe-70b1-4957-913a-ea117211d3a9" (UID: "95e3eebe-70b1-4957-913a-ea117211d3a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:47.134974 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.134926 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4" pod "95e3eebe-70b1-4957-913a-ea117211d3a9" (UID: "95e3eebe-70b1-4957-913a-ea117211d3a9"). InnerVolumeSpecName "kube-api-access-d7zp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:47.135082 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.135007 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca" pod "95e3eebe-70b1-4957-913a-ea117211d3a9" (UID: "95e3eebe-70b1-4957-913a-ea117211d3a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:47.233831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.233743 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:47.233831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.233772 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7zp4\" (UniqueName: \"kubernetes.io/projected/95e3eebe-70b1-4957-913a-ea117211d3a9-kube-api-access-d7zp4\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:47.233831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.233782 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e3eebe-70b1-4957-913a-ea117211d3a9-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:47.643009 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.642974 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:47.920099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.920019 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" event={"ID":"95e3eebe-70b1-4957-913a-ea117211d3a9","Type":"ContainerDied","Data":"ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1"} Nov 03 22:05:47.920099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.920052 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g" Nov 03 22:05:47.920099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:47.920065 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff4ab159f216ce9d6a7c44eaa0728b2bd596359bd7dd7a15f63b50d973fa3cb1" Nov 03 22:05:48.397813 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:48.397774 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:48.946170 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:48.946126 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:49.220830 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:49.220736 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:49.870062 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:49.870020 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:50.537228 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:50.537187 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:51.023961 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:51.023925 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz"] Nov 03 22:05:51.028724 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:51.028687 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d4xkz"] Nov 03 22:05:51.199163 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:51.199129 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:51.983486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:51.983434 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:52.346135 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:52.346044 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3e832f-5874-419a-a24b-0156065e42b6" path="/var/lib/kubelet/pods/8a3e832f-5874-419a-a24b-0156065e42b6/volumes" Nov 03 22:05:52.642422 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:52.642388 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:53.522280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:53.522242 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.138616 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:54.138577 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.876071 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:54.876039 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.148202 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.148173 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5"] Nov 03 22:05:55.148605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.148592 5973 cpu_manager.go:414] "containerMap: removing container" podUID="95e3eebe-70b1-4957-913a-ea117211d3a9" containerName="revoke-keys" Nov 03 22:05:55.148667 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.148606 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e3eebe-70b1-4957-913a-ea117211d3a9" containerName="revoke-keys" Nov 03 22:05:55.148709 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.148698 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e3eebe-70b1-4957-913a-ea117211d3a9" containerName="revoke-keys" Nov 03 22:05:55.152047 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.152028 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.157993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.157973 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:05:55.158649 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.158629 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:05:55.159701 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.159599 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:05:55.159701 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.159632 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:05:55.163676 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.163654 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5"] Nov 03 22:05:55.165391 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.165368 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.312345 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.312313 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.312551 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.312357 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrt5\" (UniqueName: \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.312551 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.312511 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.352607 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.352576 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:05:55.413694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.413612 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.413694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.413680 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.413875 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.413712 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrt5\" (UniqueName: \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.414248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.414219 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.416152 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.416130 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.425818 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.425797 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrt5\" (UniqueName: \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.469984 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.469957 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:55.748672 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.748644 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5"] Nov 03 22:05:55.749669 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:05:55.749644 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf435ce45_0ca3_4f97_a825_165a4568f844.slice/crio-6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b WatchSource:0}: Error finding container 6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b: Status 404 returned error can't find the container with id 6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b Nov 03 22:05:55.871987 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.871947 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.974514 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:55.974486 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" event={"ID":"f435ce45-0ca3-4f97-a825-165a4568f844","Type":"ContainerStarted","Data":"6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b"} Nov 03 22:05:56.563413 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:56.563378 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:56.868260 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:56.868172 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:56.992946 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:56.992902 5973 generic.go:356] "Generic (PLEG): container finished" podID="f435ce45-0ca3-4f97-a825-165a4568f844" containerID="76db18a41037ab45ccd346800bb1317fa5c348d924a5d17319ba5f278ecaa699" exitCode=0 Nov 03 22:05:56.993435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:56.993136 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" event={"ID":"f435ce45-0ca3-4f97-a825-165a4568f844","Type":"ContainerDied","Data":"76db18a41037ab45ccd346800bb1317fa5c348d924a5d17319ba5f278ecaa699"} Nov 03 22:05:57.497075 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:57.497034 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:58.139694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.139670 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:58.241249 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.241221 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\") pod \"f435ce45-0ca3-4f97-a825-165a4568f844\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " Nov 03 22:05:58.241450 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.241261 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\" (UniqueName: \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\") pod \"f435ce45-0ca3-4f97-a825-165a4568f844\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " Nov 03 22:05:58.241450 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.241393 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\") pod \"f435ce45-0ca3-4f97-a825-165a4568f844\" (UID: \"f435ce45-0ca3-4f97-a825-165a4568f844\") " Nov 03 22:05:58.241688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.241664 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca" pod "f435ce45-0ca3-4f97-a825-165a4568f844" (UID: "f435ce45-0ca3-4f97-a825-165a4568f844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:58.241868 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.241849 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:58.243747 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.243716 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5" pod "f435ce45-0ca3-4f97-a825-165a4568f844" (UID: "f435ce45-0ca3-4f97-a825-165a4568f844"). InnerVolumeSpecName "kube-api-access-jwrt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:58.243747 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.243734 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca" pod "f435ce45-0ca3-4f97-a825-165a4568f844" (UID: "f435ce45-0ca3-4f97-a825-165a4568f844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:58.271207 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.271138 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:58.343062 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.343029 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f435ce45-0ca3-4f97-a825-165a4568f844-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:58.343062 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:58.343063 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwrt5\" (UniqueName: \"kubernetes.io/projected/f435ce45-0ca3-4f97-a825-165a4568f844-kube-api-access-jwrt5\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:59.008962 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.008921 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" event={"ID":"f435ce45-0ca3-4f97-a825-165a4568f844","Type":"ContainerDied","Data":"6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b"} Nov 03 22:05:59.008962 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.008957 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5" Nov 03 22:05:59.009286 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.008973 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed566106b502e3a665922026c6c411832a6da61f097a31725a844b35fbe987b" Nov 03 22:05:59.073182 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.073150 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:59.756447 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.756409 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:59.975695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:05:59.975663 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:06:00.070371 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:00.070283 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:00.430583 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:00.430547 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:01.109599 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.109568 5973 scope.go:122] "RemoveContainer" containerID="3cfd3bd37da7dbedbe79c7d6a7641b11e3409c0f3c5e7bcebf92bfec3d66914c" Nov 03 22:06:01.119154 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.119131 5973 scope.go:122] "RemoveContainer" containerID="c548a3379038b1dde4d15b7a34c618d3456c29ce5cff537e164747ddc93cdfee" Nov 03 22:06:01.128632 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.128613 5973 scope.go:122] "RemoveContainer" containerID="1dcfae3b8fc5e66643dedf9705fdbc77f69e140bdd8926de52f5c21b857c78d7" Nov 03 22:06:01.137090 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.137074 5973 scope.go:122] "RemoveContainer" containerID="cdb074b3fa189cf687af6d62f63caa04323fa93a98dab49f7722311daa770f7a" Nov 03 22:06:01.145241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.145219 5973 scope.go:122] "RemoveContainer" containerID="92add4f36332dfc62b666f82eda49d4051d4be64d26bc8d7cd6b51d9ddd03af6" Nov 03 22:06:01.153640 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.153619 5973 scope.go:122] "RemoveContainer" containerID="f892b79c6c45707d78b3c94a0c54e902171ff5911c18605d3b7dd7e575d818bc" Nov 03 22:06:01.164901 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.164882 5973 scope.go:122] "RemoveContainer" containerID="4a2fc97e1f80defe9fe9cd875f1ad19e8fe8115adf769ccd22ee0fb8f6b2a3cd" Nov 03 22:06:01.173893 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.173875 5973 scope.go:122] "RemoveContainer" containerID="4ab1815edca16b7afb446d42836102e24afe32ec85db2c1bd78d2d1d8e0b6b36" Nov 03 22:06:01.181904 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.181888 5973 scope.go:122] "RemoveContainer" containerID="50942e6458902c752bffbd2bbc4a29b38c69a21d1547f5e4138d80fd7369a9e4" Nov 03 22:06:01.187669 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.187643 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:01.190589 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.190563 5973 scope.go:122] "RemoveContainer" containerID="053d54a66863c0d6d4f3714f9596d8bd48bcb7114a97f2dff6e1ce4ba2070eee" Nov 03 22:06:01.198665 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.198648 5973 scope.go:122] "RemoveContainer" containerID="dd2abc69d5f0c319f0b9d431b7bf01fac3834f3d8a9d5bf018c5803202125634" Nov 03 22:06:01.206896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.206881 5973 scope.go:122] "RemoveContainer" containerID="1aa9186a70cbdf45b74f84736aac2108437a3f772944c1434f1b5523e0959545" Nov 03 22:06:01.214548 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.214528 5973 scope.go:122] "RemoveContainer" containerID="0ccf3083fa57cceac95a34938a48cc1d2ee18fb384566e7ca4ef175413ff7108" Nov 03 22:06:01.222643 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.222627 5973 scope.go:122] "RemoveContainer" containerID="9aea7563973bf811a72b1f4c1f65cdbe653351c07a288fd31ca9d773aba06cc5" Nov 03 22:06:01.231544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.231520 5973 scope.go:122] "RemoveContainer" containerID="7977847a99b8a6f8258009b2133913cdad8027ed948ce3094f7ec23f946a6c73" Nov 03 22:06:01.242582 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.242564 5973 scope.go:122] "RemoveContainer" containerID="c1d8236055c72e1f8bc85adf9686a15765446f14dc968b2c5dc046769677425e" Nov 03 22:06:01.251958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.251937 5973 scope.go:122] "RemoveContainer" containerID="55874fc71b8ac00014b27a06304ad9b27c0be21ed28d93adf33bfccc23cd1ab7" Nov 03 22:06:01.260579 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.260560 5973 scope.go:122] "RemoveContainer" containerID="7793ac33a5b642fdbedb7a94c73ec2dabb1e2e843915b021e33c9f3eb124bd98" Nov 03 22:06:01.268676 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.268657 5973 scope.go:122] "RemoveContainer" containerID="79ab197eefcfdd75b09dbaaf24339d48c6172498dc740a169b99e349f361f7bc" Nov 03 22:06:01.874884 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:01.874848 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:02.537241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:02.537203 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:03.301551 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:03.301513 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:03.968897 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:03.968859 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:04.786257 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:04.786225 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:05.281240 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:05.281200 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:06:05.297084 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:05.297037 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:06:05.422489 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:05.422431 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:05.701985 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:05.701953 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:06.141045 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:06.140955 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:06.374491 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:06.374410 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5"] Nov 03 22:06:06.377072 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:06.377050 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-94e879-cgmt5"] Nov 03 22:06:06.422034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:06.422004 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:06.894246 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:06.894216 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:07.890236 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:07.890196 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:06:08.348436 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:08.348404 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f435ce45-0ca3-4f97-a825-165a4568f844" path="/var/lib/kubelet/pods/f435ce45-0ca3-4f97-a825-165a4568f844/volumes" Nov 03 22:06:10.282127 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:10.282091 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" containerID="cri-o://4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8" gracePeriod=25 Nov 03 22:06:10.298682 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:10.298650 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" containerID="cri-o://f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a" gracePeriod=25 Nov 03 22:06:10.580992 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:10.580913 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:06:10.580992 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:10.580948 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:06:13.209877 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:13.209829 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:13.497182 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:13.497092 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:13.519411 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:13.519381 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:06:13.519681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:13.519655 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" containerID="cri-o://f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013" gracePeriod=30 Nov 03 22:06:13.772098 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:13.772004 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:14.399961 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:14.399925 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:06:14.763497 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.763405 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm"] Nov 03 22:06:14.770939 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.770910 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wvghm"] Nov 03 22:06:14.789104 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.789073 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k"] Nov 03 22:06:14.789541 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.789523 5973 cpu_manager.go:414] "containerMap: removing container" podUID="f435ce45-0ca3-4f97-a825-165a4568f844" containerName="revoke-keys" Nov 03 22:06:14.789620 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.789543 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f435ce45-0ca3-4f97-a825-165a4568f844" containerName="revoke-keys" Nov 03 22:06:14.789685 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.789671 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="f435ce45-0ca3-4f97-a825-165a4568f844" containerName="revoke-keys" Nov 03 22:06:14.796009 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.795990 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:14.800737 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.800713 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:06:14.800737 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.800725 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:06:14.800893 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.800881 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:06:14.804192 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.800993 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:06:14.804192 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.803108 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k"] Nov 03 22:06:14.899756 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.899726 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g568x\" (UniqueName: \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:14.899912 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.899791 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:14.899912 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.899859 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:14.996122 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:14.996090 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" podUID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" containerName="main" containerID="cri-o://409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e" gracePeriod=45 Nov 03 22:06:15.000607 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.000582 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g568x\" (UniqueName: \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.000687 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.000667 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.000722 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.000696 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.001279 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.001258 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.003402 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.003378 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.013011 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.012991 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g568x\" (UniqueName: \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\") pod \"maas-api-revoke-sub-3d4156f1-qgf7k\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.111102 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.111074 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:15.279646 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.279588 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:06:15.303039 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303015 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303059 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303103 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303136 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303322 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303257 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\" (UniqueName: \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303322 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303288 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\") pod \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\" (UID: \"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a\") " Nov 03 22:06:15.303437 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303384 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:15.303437 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303419 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:15.303563 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303484 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:15.303692 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303675 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:15.303743 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303697 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:15.303743 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.303712 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:15.306272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.306241 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:15.306392 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.306354 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "kube-api-access-mbkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:15.306516 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.306497 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm" pod "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" (UID: "64a4fdf8-0ea5-4f89-a9c6-687f52487a2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:15.391871 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.391844 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k"] Nov 03 22:06:15.393619 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:06:15.393595 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453358f7_57fd_4057_ad7f_05deba163909.slice/crio-9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025 WatchSource:0}: Error finding container 9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025: Status 404 returned error can't find the container with id 9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025 Nov 03 22:06:15.404959 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.404936 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkw7\" (UniqueName: \"kubernetes.io/projected/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-kube-api-access-mbkw7\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:15.405266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.404964 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:15.405266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:15.404977 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:16.114880 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.114849 5973 generic.go:356] "Generic (PLEG): container finished" podID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" containerID="409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e" exitCode=0 Nov 03 22:06:16.115055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.114924 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" Nov 03 22:06:16.115055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.114940 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" event={"ID":"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a","Type":"ContainerDied","Data":"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e"} Nov 03 22:06:16.115055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.114986 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw" event={"ID":"64a4fdf8-0ea5-4f89-a9c6-687f52487a2a","Type":"ContainerDied","Data":"93d47398bc84b06435da9489d38af70ee34c63bb9588b8e295ba63629ecefb4e"} Nov 03 22:06:16.115055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.115009 5973 scope.go:122] "RemoveContainer" containerID="409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e" Nov 03 22:06:16.116484 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.116426 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" event={"ID":"453358f7-57fd-4057-ad7f-05deba163909","Type":"ContainerStarted","Data":"782d401cec88e47e8479c69c8d6b0f91b5dd59b891d0b8255aacc916de6eba0f"} Nov 03 22:06:16.116547 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.116504 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" event={"ID":"453358f7-57fd-4057-ad7f-05deba163909","Type":"ContainerStarted","Data":"9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025"} Nov 03 22:06:16.124896 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.124877 5973 scope.go:122] "RemoveContainer" containerID="409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e" Nov 03 22:06:16.125171 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:16.125151 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e\": container with ID starting with 409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e not found: ID does not exist" containerID="409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e" Nov 03 22:06:16.125219 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.125179 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e"} err="failed to get container status \"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e\": rpc error: code = NotFound desc = could not find container \"409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e\": container with ID starting with 409ef244cd04b0f4c10e1609c648c2bdc1d950a195a8aa9a14b3cd3e976c190e not found: ID does not exist" Nov 03 22:06:16.138065 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.138028 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" podStartSLOduration=2.138017235 podStartE2EDuration="2.138017235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:16.136046275 +0000 UTC m=+1215.910896123" watchObservedRunningTime="2026-11-03 22:06:16.138017235 +0000 UTC m=+1215.912867082" Nov 03 22:06:16.150659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.150635 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:06:16.155069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.155049 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-94e879/simulator-single-main-94e879-kserve-d57878c54-mgpfw"] Nov 03 22:06:16.344450 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.344424 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" path="/var/lib/kubelet/pods/64a4fdf8-0ea5-4f89-a9c6-687f52487a2a/volumes" Nov 03 22:06:16.344845 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:16.344831 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f374b9-05c0-4eb1-a8a5-15ada787e425" path="/var/lib/kubelet/pods/a8f374b9-05c0-4eb1-a8a5-15ada787e425/volumes" Nov 03 22:06:17.126096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:17.126063 5973 generic.go:356] "Generic (PLEG): container finished" podID="453358f7-57fd-4057-ad7f-05deba163909" containerID="782d401cec88e47e8479c69c8d6b0f91b5dd59b891d0b8255aacc916de6eba0f" exitCode=0 Nov 03 22:06:17.126504 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:17.126155 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" event={"ID":"453358f7-57fd-4057-ad7f-05deba163909","Type":"ContainerDied","Data":"782d401cec88e47e8479c69c8d6b0f91b5dd59b891d0b8255aacc916de6eba0f"} Nov 03 22:06:18.267701 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.267671 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:18.333300 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.333271 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\") pod \"453358f7-57fd-4057-ad7f-05deba163909\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " Nov 03 22:06:18.333476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.333310 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\") pod \"453358f7-57fd-4057-ad7f-05deba163909\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " Nov 03 22:06:18.333476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.333344 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\" (UniqueName: \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\") pod \"453358f7-57fd-4057-ad7f-05deba163909\" (UID: \"453358f7-57fd-4057-ad7f-05deba163909\") " Nov 03 22:06:18.333690 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.333669 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca" pod "453358f7-57fd-4057-ad7f-05deba163909" (UID: "453358f7-57fd-4057-ad7f-05deba163909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:18.335622 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.335603 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca" pod "453358f7-57fd-4057-ad7f-05deba163909" (UID: "453358f7-57fd-4057-ad7f-05deba163909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:18.335746 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.335729 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x" pod "453358f7-57fd-4057-ad7f-05deba163909" (UID: "453358f7-57fd-4057-ad7f-05deba163909"). InnerVolumeSpecName "kube-api-access-g568x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:18.434481 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.434421 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453358f7-57fd-4057-ad7f-05deba163909-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.434481 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.434480 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453358f7-57fd-4057-ad7f-05deba163909-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.434680 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.434499 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g568x\" (UniqueName: \"kubernetes.io/projected/453358f7-57fd-4057-ad7f-05deba163909-kube-api-access-g568x\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.758400 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.758372 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:06:18.838982 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.838951 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.838999 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839046 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839085 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839124 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839224 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839257 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839295 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839329 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:18.839363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839341 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\") pod \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\" (UID: \"ccb0ef56-489f-454a-8cf5-29fffaaf7671\") " Nov 03 22:06:18.839649 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839362 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:18.839649 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839606 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:18.839770 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839747 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:18.839824 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839773 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:18.839882 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839844 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.839882 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839862 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.839882 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839877 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.840012 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839891 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.840012 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.839902 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.841417 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.841390 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:06:18.841704 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.841683 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "kube-api-access-bwdj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:18.841845 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.841827 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:18.841894 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.841839 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token" pod "ccb0ef56-489f-454a-8cf5-29fffaaf7671" (UID: "ccb0ef56-489f-454a-8cf5-29fffaaf7671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:18.941299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.941237 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.941299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.941259 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.941299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.941268 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb0ef56-489f-454a-8cf5-29fffaaf7671-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:18.941299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:18.941277 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdj7\" (UniqueName: \"kubernetes.io/projected/ccb0ef56-489f-454a-8cf5-29fffaaf7671-kube-api-access-bwdj7\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:19.139505 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.139468 5973 generic.go:356] "Generic (PLEG): container finished" podID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerID="f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013" exitCode=0 Nov 03 22:06:19.139689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.139516 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" Nov 03 22:06:19.139689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.139550 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerDied","Data":"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013"} Nov 03 22:06:19.139689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.139602 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv" event={"ID":"ccb0ef56-489f-454a-8cf5-29fffaaf7671","Type":"ContainerDied","Data":"2dff716d2f7fdeccb3b6c26c3bd2d32f00e42b4111116ef864a105ad0bd534d8"} Nov 03 22:06:19.139689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.139628 5973 scope.go:122] "RemoveContainer" containerID="f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013" Nov 03 22:06:19.141055 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.141040 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" Nov 03 22:06:19.141151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.141112 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k" event={"ID":"453358f7-57fd-4057-ad7f-05deba163909","Type":"ContainerDied","Data":"9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025"} Nov 03 22:06:19.141194 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.141159 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9efc521f487595126ca53a7063e639d7a564164d627aa80115e7a8ac16b06025" Nov 03 22:06:19.149582 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.149567 5973 scope.go:122] "RemoveContainer" containerID="d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4" Nov 03 22:06:19.158201 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.158185 5973 scope.go:122] "RemoveContainer" containerID="f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013" Nov 03 22:06:19.158450 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:19.158431 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013\": container with ID starting with f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013 not found: ID does not exist" containerID="f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013" Nov 03 22:06:19.158510 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.158473 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013"} err="failed to get container status \"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013\": rpc error: code = NotFound desc = could not find container \"f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013\": container with ID starting with f741a118f051317d778b9b496f486720f81f31cdd18f489888fc1f7dc73f7013 not found: ID does not exist" Nov 03 22:06:19.158510 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.158492 5973 scope.go:122] "RemoveContainer" containerID="d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4" Nov 03 22:06:19.158722 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:19.158708 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4\": container with ID starting with d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4 not found: ID does not exist" containerID="d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4" Nov 03 22:06:19.158762 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.158724 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4"} err="failed to get container status \"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4\": rpc error: code = NotFound desc = could not find container \"d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4\": container with ID starting with d0e888c07f930a4ac9dd14cf1475a864129e5939010bf2a083b5099dc7507dd4 not found: ID does not exist" Nov 03 22:06:19.163741 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.163719 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:06:19.170541 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:19.170521 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-94e879-openshift-default-77c678fcc-5wbgv"] Nov 03 22:06:20.346769 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:20.346736 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" path="/var/lib/kubelet/pods/ccb0ef56-489f-454a-8cf5-29fffaaf7671/volumes" Nov 03 22:06:20.580688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:20.580656 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:06:20.580688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:20.580678 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:06:25.553404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:25.553362 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:06:29.066789 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:29.066757 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:06:30.550929 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.550866 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:06:30.578645 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.578620 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:06:30.666899 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.666867 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\" (UniqueName: \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\") pod \"7c2951cf-e822-449b-948b-4f1c858ebd7c\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " Nov 03 22:06:30.666899 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.666903 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\" (UniqueName: \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\") pod \"3278887b-3642-4447-8686-b571ca2eebe6\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " Nov 03 22:06:30.667147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.666964 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\") pod \"7c2951cf-e822-449b-948b-4f1c858ebd7c\" (UID: \"7c2951cf-e822-449b-948b-4f1c858ebd7c\") " Nov 03 22:06:30.667147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.667113 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\") pod \"3278887b-3642-4447-8686-b571ca2eebe6\" (UID: \"3278887b-3642-4447-8686-b571ca2eebe6\") " Nov 03 22:06:30.667436 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.667409 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume" pod "7c2951cf-e822-449b-948b-4f1c858ebd7c" (UID: "7c2951cf-e822-449b-948b-4f1c858ebd7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:30.667574 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.667439 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume" pod "3278887b-3642-4447-8686-b571ca2eebe6" (UID: "3278887b-3642-4447-8686-b571ca2eebe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:30.669212 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.669196 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj" pod "7c2951cf-e822-449b-948b-4f1c858ebd7c" (UID: "7c2951cf-e822-449b-948b-4f1c858ebd7c"). InnerVolumeSpecName "kube-api-access-4c2rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:30.669273 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.669241 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm" pod "3278887b-3642-4447-8686-b571ca2eebe6" (UID: "3278887b-3642-4447-8686-b571ca2eebe6"). InnerVolumeSpecName "kube-api-access-gmjnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:30.768861 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.768818 5973 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2951cf-e822-449b-948b-4f1c858ebd7c-plugins-config-volume\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.768861 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.768856 5973 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278887b-3642-4447-8686-b571ca2eebe6-plugins-config-volume\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.768861 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.768866 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2rj\" (UniqueName: \"kubernetes.io/projected/7c2951cf-e822-449b-948b-4f1c858ebd7c-kube-api-access-4c2rj\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:30.768861 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:30.768875 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjnm\" (UniqueName: \"kubernetes.io/projected/3278887b-3642-4447-8686-b571ca2eebe6-kube-api-access-gmjnm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:31.256710 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.256673 5973 generic.go:356] "Generic (PLEG): container finished" podID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerID="4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8" exitCode=0 Nov 03 22:06:31.256910 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.256731 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" Nov 03 22:06:31.256910 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.256759 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" event={"ID":"7c2951cf-e822-449b-948b-4f1c858ebd7c","Type":"ContainerDied","Data":"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8"} Nov 03 22:06:31.256910 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.256844 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv" event={"ID":"7c2951cf-e822-449b-948b-4f1c858ebd7c","Type":"ContainerDied","Data":"b8d9baa4beba0e8a957fcb520e8ab56078af092d4f78798619c2f07db6b8b7cf"} Nov 03 22:06:31.256910 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.256870 5973 scope.go:122] "RemoveContainer" containerID="4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8" Nov 03 22:06:31.258245 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.258221 5973 generic.go:356] "Generic (PLEG): container finished" podID="3278887b-3642-4447-8686-b571ca2eebe6" containerID="f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a" exitCode=0 Nov 03 22:06:31.258341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.258329 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" Nov 03 22:06:31.258411 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.258331 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" event={"ID":"3278887b-3642-4447-8686-b571ca2eebe6","Type":"ContainerDied","Data":"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a"} Nov 03 22:06:31.258499 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.258429 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq" event={"ID":"3278887b-3642-4447-8686-b571ca2eebe6","Type":"ContainerDied","Data":"912bf974ec4fb12f93c9a3dcd42ed75119bbd10e1c2b4a8ec146099742e7be56"} Nov 03 22:06:31.268248 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.268227 5973 scope.go:122] "RemoveContainer" containerID="4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8" Nov 03 22:06:31.268555 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:31.268531 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8\": container with ID starting with 4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8 not found: ID does not exist" containerID="4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8" Nov 03 22:06:31.268612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.268569 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8"} err="failed to get container status \"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8\": rpc error: code = NotFound desc = could not find container \"4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8\": container with ID starting with 4214df708434312d5f3094446c85294bb4d643a7317e6fd45f7757bfd14548d8 not found: ID does not exist" Nov 03 22:06:31.268612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.268587 5973 scope.go:122] "RemoveContainer" containerID="f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a" Nov 03 22:06:31.276848 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.276834 5973 scope.go:122] "RemoveContainer" containerID="f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a" Nov 03 22:06:31.277135 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:31.277112 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a\": container with ID starting with f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a not found: ID does not exist" containerID="f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a" Nov 03 22:06:31.277179 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.277143 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a"} err="failed to get container status \"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a\": rpc error: code = NotFound desc = could not find container \"f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a\": container with ID starting with f9b8e61f07a2793d2629de4e610aa582d44d8140530f73f8a5ffab0f8d552c4a not found: ID does not exist" Nov 03 22:06:31.281096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.281073 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:06:31.283950 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.283930 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-94e879-58b7d84bb252cv"] Nov 03 22:06:31.295297 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.295276 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:06:31.299485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:31.299447 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-94e879-74c8bpgcfq"] Nov 03 22:06:32.343904 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:32.343877 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3278887b-3642-4447-8686-b571ca2eebe6" path="/var/lib/kubelet/pods/3278887b-3642-4447-8686-b571ca2eebe6/volumes" Nov 03 22:06:32.344290 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:32.344214 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" path="/var/lib/kubelet/pods/7c2951cf-e822-449b-948b-4f1c858ebd7c/volumes" Nov 03 22:06:32.641274 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:32.641185 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:32.941073 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:32.941037 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:36.272101 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:36.272063 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:37.523096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:37.523047 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:38.023694 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:38.023658 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8"] Nov 03 22:06:38.026256 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:38.026228 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5xhf8"] Nov 03 22:06:38.344509 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:38.344423 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2db19b0-9030-4ded-9981-d88032c92670" path="/var/lib/kubelet/pods/a2db19b0-9030-4ded-9981-d88032c92670/volumes" Nov 03 22:06:43.137799 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:43.137759 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:44.150751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:44.150717 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:44.398907 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:06:44.398865 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:06:44.428728 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:44.428635 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:56.862517 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:56.862482 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:57.022333 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:57.022292 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk"] Nov 03 22:06:57.025082 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:57.025059 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gqlzk"] Nov 03 22:06:58.344816 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:06:58.344782 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b" path="/var/lib/kubelet/pods/a3ce4a12-34da-4a25-bf3e-d09ffa6e7b4b/volumes" Nov 03 22:07:01.367305 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:01.367275 5973 scope.go:122] "RemoveContainer" containerID="884e0fdaff98a8d36976c655914e22670c9f74c6fbc1bb4a7d3806ece77c846f" Nov 03 22:07:01.376372 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:01.376352 5973 scope.go:122] "RemoveContainer" containerID="df2293cca2c05c277446c2eff95d358736485af083e18248af699c1a364e1025" Nov 03 22:07:01.509852 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:01.509815 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:08.023804 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:08.023773 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697"] Nov 03 22:07:08.026030 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:08.026006 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f8697"] Nov 03 22:07:08.344918 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:08.344844 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb" path="/var/lib/kubelet/pods/6dd06096-8acb-4d3c-8cf5-4fc8c2854fdb/volumes" Nov 03 22:07:14.400369 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:07:14.400291 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:07:34.023954 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:34.023913 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw"] Nov 03 22:07:34.027935 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:34.027908 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l4tqw"] Nov 03 22:07:34.344989 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:34.344912 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf541d42-f385-4e51-9788-265dc68719d2" path="/var/lib/kubelet/pods/cf541d42-f385-4e51-9788-265dc68719d2/volumes" Nov 03 22:07:44.023082 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:44.023045 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4"] Nov 03 22:07:44.025750 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:44.025726 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58hn4"] Nov 03 22:07:44.344353 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:44.344281 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e185d7f-b2cd-433f-b6f1-d0b506966e3e" path="/var/lib/kubelet/pods/4e185d7f-b2cd-433f-b6f1-d0b506966e3e/volumes" Nov 03 22:07:44.399369 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:07:44.399335 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:07:53.250847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:53.250816 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:07:56.833154 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:56.833105 5973 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:57.223170 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223136 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz"] Nov 03 22:07:57.223646 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223629 5973 cpu_manager.go:414] "containerMap: removing container" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223648 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223661 5973 cpu_manager.go:414] "containerMap: removing container" podUID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" containerName="main" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223668 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" containerName="main" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223688 5973 cpu_manager.go:414] "containerMap: removing container" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223695 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223711 5973 cpu_manager.go:414] "containerMap: removing container" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223718 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223731 5973 cpu_manager.go:414] "containerMap: removing container" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" Nov 03 22:07:57.223751 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223738 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223755 5973 cpu_manager.go:414] "containerMap: removing container" podUID="453358f7-57fd-4057-ad7f-05deba163909" containerName="revoke-keys" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223762 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453358f7-57fd-4057-ad7f-05deba163909" containerName="revoke-keys" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223865 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223877 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a4fdf8-0ea5-4f89-a9c6-687f52487a2a" containerName="main" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223887 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="3278887b-3642-4447-8686-b571ca2eebe6" containerName="payload-pre-processing" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223897 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2951cf-e822-449b-948b-4f1c858ebd7c" containerName="payload-processing" Nov 03 22:07:57.224178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.223907 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="453358f7-57fd-4057-ad7f-05deba163909" containerName="revoke-keys" Nov 03 22:07:57.227098 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.227079 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.231925 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.231907 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:07:57.231925 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.231916 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:07:57.232069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.231911 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:07:57.232069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.231916 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:07:57.238403 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.238379 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz"] Nov 03 22:07:57.248964 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.248943 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.249070 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.249005 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.249114 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.249068 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6x4\" (UniqueName: \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.350137 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.350104 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.350303 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.350200 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.350303 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.350249 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6x4\" (UniqueName: \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.350875 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.350855 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.352715 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.352691 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.362313 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.362293 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6x4\" (UniqueName: \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\") pod \"maas-api-revoke-sub-d576e68b-nm2cz\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.541321 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.541243 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:57.817511 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:57.817486 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz"] Nov 03 22:07:57.819286 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:07:57.819259 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29658059_0d2b_486b_84e8_506b4985c819.slice/crio-e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172 WatchSource:0}: Error finding container e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172: Status 404 returned error can't find the container with id e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172 Nov 03 22:07:58.753531 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:58.753497 5973 generic.go:356] "Generic (PLEG): container finished" podID="29658059-0d2b-486b-84e8-506b4985c819" containerID="e294d55b4d2c366186274267041c6d24094592e0628b5f0c8671e54be0f6458a" exitCode=0 Nov 03 22:07:58.753531 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:58.753531 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" event={"ID":"29658059-0d2b-486b-84e8-506b4985c819","Type":"ContainerDied","Data":"e294d55b4d2c366186274267041c6d24094592e0628b5f0c8671e54be0f6458a"} Nov 03 22:07:58.753970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:58.753568 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" event={"ID":"29658059-0d2b-486b-84e8-506b4985c819","Type":"ContainerStarted","Data":"e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172"} Nov 03 22:07:59.887099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.887075 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:07:59.975286 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.975258 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\") pod \"29658059-0d2b-486b-84e8-506b4985c819\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " Nov 03 22:07:59.975433 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.975370 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\" (UniqueName: \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\") pod \"29658059-0d2b-486b-84e8-506b4985c819\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " Nov 03 22:07:59.975515 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.975498 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\") pod \"29658059-0d2b-486b-84e8-506b4985c819\" (UID: \"29658059-0d2b-486b-84e8-506b4985c819\") " Nov 03 22:07:59.975821 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.975792 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca" pod "29658059-0d2b-486b-84e8-506b4985c819" (UID: "29658059-0d2b-486b-84e8-506b4985c819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:59.977500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.977479 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca" pod "29658059-0d2b-486b-84e8-506b4985c819" (UID: "29658059-0d2b-486b-84e8-506b4985c819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:59.977582 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:07:59.977533 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4" pod "29658059-0d2b-486b-84e8-506b4985c819" (UID: "29658059-0d2b-486b-84e8-506b4985c819"). InnerVolumeSpecName "kube-api-access-vf6x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:00.076280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.076194 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29658059-0d2b-486b-84e8-506b4985c819-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:00.076280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.076227 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29658059-0d2b-486b-84e8-506b4985c819-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:00.076280 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.076238 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6x4\" (UniqueName: \"kubernetes.io/projected/29658059-0d2b-486b-84e8-506b4985c819-kube-api-access-vf6x4\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:00.766681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.766651 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" Nov 03 22:08:00.766885 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.766650 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz" event={"ID":"29658059-0d2b-486b-84e8-506b4985c819","Type":"ContainerDied","Data":"e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172"} Nov 03 22:08:00.766885 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:00.766769 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5716c14a2687bcf3e63d380b516bbf75d5b5b240a57499af2aedda135564172" Nov 03 22:08:01.418830 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:01.418795 5973 scope.go:122] "RemoveContainer" containerID="c2b45eaf00bd8d595b29494726dbfca2f625f67aeaeac39e3d4c9a19ce38b7dd" Nov 03 22:08:01.427907 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:01.427884 5973 scope.go:122] "RemoveContainer" containerID="3f776ee15042a3eb3a4684ce8ad6c1e794b25f32c9da739303d0be509678d3a3" Nov 03 22:08:04.024369 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:04.024330 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9"] Nov 03 22:08:04.026332 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:04.026306 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lgnt9"] Nov 03 22:08:04.344244 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:04.344172 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd272b7a-d9ea-4fa5-9e31-6240195b1ace" path="/var/lib/kubelet/pods/fd272b7a-d9ea-4fa5-9e31-6240195b1ace/volumes" Nov 03 22:08:08.038948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:08.038914 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:08:13.292015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.291983 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9"] Nov 03 22:08:13.292404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.292366 5973 cpu_manager.go:414] "containerMap: removing container" podUID="29658059-0d2b-486b-84e8-506b4985c819" containerName="revoke-keys" Nov 03 22:08:13.292404 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.292380 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29658059-0d2b-486b-84e8-506b4985c819" containerName="revoke-keys" Nov 03 22:08:13.292518 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.292503 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb0ef56-489f-454a-8cf5-29fffaaf7671" containerName="istio-proxy" Nov 03 22:08:13.292555 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.292522 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="29658059-0d2b-486b-84e8-506b4985c819" containerName="revoke-keys" Nov 03 22:08:13.296795 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.296780 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.301334 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.301311 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:08:13.301482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.301390 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:08:13.301482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.301391 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:08:13.301482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.301397 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:08:13.306340 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.306314 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9"] Nov 03 22:08:13.424530 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.424502 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.424706 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.424553 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62ds\" (UniqueName: \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.424706 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.424684 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.525528 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.525477 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62ds\" (UniqueName: \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.525748 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.525591 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.525748 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.525624 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.526193 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.526176 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.528077 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.528058 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.537620 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.537598 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62ds\" (UniqueName: \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\") pod \"maas-api-revoke-sub-d2bc93bc-ttkw9\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.611272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.611201 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:13.889837 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:13.889809 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9"] Nov 03 22:08:13.891683 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:08:13.891657 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815b129a_64f1_4520_8039_97e095097a4a.slice/crio-acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc WatchSource:0}: Error finding container acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc: Status 404 returned error can't find the container with id acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc Nov 03 22:08:14.399430 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:08:14.399397 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:08:14.850551 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:14.850518 5973 generic.go:356] "Generic (PLEG): container finished" podID="815b129a-64f1-4520-8039-97e095097a4a" containerID="31f31f0d3331c14b78d8ada29b90b2838a992e2202a63fd30c84887c9fa0f688" exitCode=0 Nov 03 22:08:14.850723 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:14.850598 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" event={"ID":"815b129a-64f1-4520-8039-97e095097a4a","Type":"ContainerDied","Data":"31f31f0d3331c14b78d8ada29b90b2838a992e2202a63fd30c84887c9fa0f688"} Nov 03 22:08:14.850723 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:14.850670 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" event={"ID":"815b129a-64f1-4520-8039-97e095097a4a","Type":"ContainerStarted","Data":"acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc"} Nov 03 22:08:15.986948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:15.986922 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:16.046972 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.046947 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\" (UniqueName: \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\") pod \"815b129a-64f1-4520-8039-97e095097a4a\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " Nov 03 22:08:16.047110 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.047041 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\") pod \"815b129a-64f1-4520-8039-97e095097a4a\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " Nov 03 22:08:16.047110 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.047069 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\") pod \"815b129a-64f1-4520-8039-97e095097a4a\" (UID: \"815b129a-64f1-4520-8039-97e095097a4a\") " Nov 03 22:08:16.047407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.047378 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca" pod "815b129a-64f1-4520-8039-97e095097a4a" (UID: "815b129a-64f1-4520-8039-97e095097a4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:16.049207 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.049179 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds" pod "815b129a-64f1-4520-8039-97e095097a4a" (UID: "815b129a-64f1-4520-8039-97e095097a4a"). InnerVolumeSpecName "kube-api-access-t62ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:16.049293 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.049241 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca" pod "815b129a-64f1-4520-8039-97e095097a4a" (UID: "815b129a-64f1-4520-8039-97e095097a4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:16.148390 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.148369 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t62ds\" (UniqueName: \"kubernetes.io/projected/815b129a-64f1-4520-8039-97e095097a4a-kube-api-access-t62ds\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:16.148390 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.148391 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815b129a-64f1-4520-8039-97e095097a4a-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:16.148571 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.148401 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815b129a-64f1-4520-8039-97e095097a4a-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:16.862030 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.861999 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" Nov 03 22:08:16.862215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.862037 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9" event={"ID":"815b129a-64f1-4520-8039-97e095097a4a","Type":"ContainerDied","Data":"acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc"} Nov 03 22:08:16.862215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:16.862092 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb9df1e874f70cf003ad65c0e3dd8f6144a4087b5c16b12b65aa380d788d7dc" Nov 03 22:08:22.037110 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:22.037074 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm"] Nov 03 22:08:22.041564 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:22.041539 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4hkxm"] Nov 03 22:08:22.344822 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:22.344739 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df87ea2e-03ca-4f7e-8f08-0107074dd1ed" path="/var/lib/kubelet/pods/df87ea2e-03ca-4f7e-8f08-0107074dd1ed/volumes" Nov 03 22:08:25.170018 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.169970 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:08:25.170527 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.170400 5973 cpu_manager.go:414] "containerMap: removing container" podUID="815b129a-64f1-4520-8039-97e095097a4a" containerName="revoke-keys" Nov 03 22:08:25.170527 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.170413 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815b129a-64f1-4520-8039-97e095097a4a" containerName="revoke-keys" Nov 03 22:08:25.170527 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.170491 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="815b129a-64f1-4520-8039-97e095097a4a" containerName="revoke-keys" Nov 03 22:08:25.173871 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.173853 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.181535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.181514 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-26be95-openshift-default-dockercfg-z85vj\"" Nov 03 22:08:25.187297 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.187274 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:08:25.235065 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235036 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235208 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235088 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235208 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235130 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235206 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235267 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmvl\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235318 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235375 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235415 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.235520 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.235507 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336390 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336351 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmvl\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336611 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336430 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336611 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336500 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336611 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336532 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336611 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336569 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336614 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336637 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336661 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.336847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.336725 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.337274 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.337249 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.337594 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.337298 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.337758 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.337405 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.337758 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.337535 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.338091 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.337613 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.339660 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.339621 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.340157 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.340118 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.349715 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.349661 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmvl\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.349848 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.349831 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\") pod \"e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.488052 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.487965 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:25.769713 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.769434 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:08:25.772138 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:08:25.772105 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0077f5_7c4e_44fe_b2bc_59b6ab837df5.slice/crio-13faf447e280e43aa7775c6d14027ca5443058d6f4cf3041c750fd0011dd7e76 WatchSource:0}: Error finding container 13faf447e280e43aa7775c6d14027ca5443058d6f4cf3041c750fd0011dd7e76: Status 404 returned error can't find the container with id 13faf447e280e43aa7775c6d14027ca5443058d6f4cf3041c750fd0011dd7e76 Nov 03 22:08:25.774423 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.774381 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:25.774557 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.774490 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:25.774557 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.774534 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:25.914293 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.914260 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerStarted","Data":"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908"} Nov 03 22:08:25.914503 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.914316 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerStarted","Data":"13faf447e280e43aa7775c6d14027ca5443058d6f4cf3041c750fd0011dd7e76"} Nov 03 22:08:25.937186 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:25.937102 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podStartSLOduration=0.93708519 podStartE2EDuration="937.08519ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:25.936020342 +0000 UTC m=+1345.710870194" watchObservedRunningTime="2026-11-03 22:08:25.93708519 +0000 UTC m=+1345.711935040" Nov 03 22:08:26.488250 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:26.488219 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:26.493290 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:26.493267 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:26.919780 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:26.919752 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:26.920599 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:26.920579 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:29.422507 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.422160 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:08:29.426268 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.426242 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.435449 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.435429 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-26be95\"" Nov 03 22:08:29.438513 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.438491 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:08:29.457505 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.457479 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:08:29.461488 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.461452 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.473273 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.473248 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:08:29.541045 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.541018 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c7ac596e-dockercfg-f4nwd\"" Nov 03 22:08:29.579662 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.579618 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnqn\" (UniqueName: \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.579978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.579955 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjcr\" (UniqueName: \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.580109 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.580094 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.580406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.580365 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.681684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.681600 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnqn\" (UniqueName: \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.681684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.681662 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjcr\" (UniqueName: \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.681684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.681683 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.681935 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.681737 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.682440 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.682418 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.682553 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.682535 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.694202 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.694174 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjcr\" (UniqueName: \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\") pod \"payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:29.694202 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.694198 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnqn\" (UniqueName: \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\") pod \"payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.742324 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.742299 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:29.775689 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:29.775655 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:30.027069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.027040 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:08:30.028537 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:08:30.028506 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102f3380_0727_4b3e_8c76_20eafa08c763.slice/crio-2264de87d14ec72fe6b998ac90c2f1a6537e49a328bacf7964bd1a3efc46839c WatchSource:0}: Error finding container 2264de87d14ec72fe6b998ac90c2f1a6537e49a328bacf7964bd1a3efc46839c: Status 404 returned error can't find the container with id 2264de87d14ec72fe6b998ac90c2f1a6537e49a328bacf7964bd1a3efc46839c Nov 03 22:08:30.076153 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.076130 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:08:30.078217 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:08:30.078191 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acb6a93_f1aa_42dc_b745_60f834a89e2c.slice/crio-d4abeff8efe5e53afa15fff8e95024d78787abd22185a47ced01954372aa8528 WatchSource:0}: Error finding container d4abeff8efe5e53afa15fff8e95024d78787abd22185a47ced01954372aa8528: Status 404 returned error can't find the container with id d4abeff8efe5e53afa15fff8e95024d78787abd22185a47ced01954372aa8528 Nov 03 22:08:30.942388 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.942353 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" event={"ID":"102f3380-0727-4b3e-8c76-20eafa08c763","Type":"ContainerStarted","Data":"82d81fed1c6761fc5cc04a5d3aba4124ae7065394ee314543c6eb54b548418e4"} Nov 03 22:08:30.942877 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.942407 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" event={"ID":"102f3380-0727-4b3e-8c76-20eafa08c763","Type":"ContainerStarted","Data":"2264de87d14ec72fe6b998ac90c2f1a6537e49a328bacf7964bd1a3efc46839c"} Nov 03 22:08:30.942877 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.942469 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:30.944149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.944124 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" event={"ID":"3acb6a93-f1aa-42dc-b745-60f834a89e2c","Type":"ContainerStarted","Data":"c337128a93a981fd1bd801c0ff7c82b4c5f726f619c33f3f3a7a00cacf6eef3e"} Nov 03 22:08:30.944258 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.944160 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" event={"ID":"3acb6a93-f1aa-42dc-b745-60f834a89e2c","Type":"ContainerStarted","Data":"d4abeff8efe5e53afa15fff8e95024d78787abd22185a47ced01954372aa8528"} Nov 03 22:08:30.944258 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.944179 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:30.962141 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.962097 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" podStartSLOduration=1.962085165 podStartE2EDuration="1.962085165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:30.959698802 +0000 UTC m=+1350.734548646" watchObservedRunningTime="2026-11-03 22:08:30.962085165 +0000 UTC m=+1350.736935013" Nov 03 22:08:30.979227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:30.979188 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" podStartSLOduration=1.979176721 podStartE2EDuration="1.979176721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:30.976962798 +0000 UTC m=+1350.751812646" watchObservedRunningTime="2026-11-03 22:08:30.979176721 +0000 UTC m=+1350.754026567" Nov 03 22:08:31.648003 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:31.647971 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:08:41.028907 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.028870 5973 generic.go:356] "Generic (PLEG): container finished" podID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerID="3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908" exitCode=0 Nov 03 22:08:41.029316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.028940 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerDied","Data":"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908"} Nov 03 22:08:41.029316 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.028997 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9"] Nov 03 22:08:41.029397 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.029327 5973 scope.go:122] "RemoveContainer" containerID="3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908" Nov 03 22:08:41.030550 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.030516 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:41.030681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.030598 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:41.030681 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.030659 5973 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 22:08:41.033206 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.033188 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tsmg9"] Nov 03 22:08:41.920266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.920229 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:41.953991 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.953964 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:08:41.954156 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:41.954021 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:08:42.041652 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:42.041619 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerStarted","Data":"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f"} Nov 03 22:08:42.344636 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:42.344558 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2266f37b-4d68-4b0c-b920-b964f2a4b1a1" path="/var/lib/kubelet/pods/2266f37b-4d68-4b0c-b920-b964f2a4b1a1/volumes" Nov 03 22:08:42.489106 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:42.489075 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:42.490116 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:42.490095 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:08:42.490172 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:42.490151 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 22:08:43.488328 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:43.488293 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:08:43.488764 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:43.488346 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 22:08:44.406727 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:08:44.406689 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:08:44.489001 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.488965 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:08:44.489396 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.489023 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 22:08:44.916264 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.916231 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:08:44.921281 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.921257 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.931181 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.931152 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:08:44.932724 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.932703 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26be95\"/\"openshift-service-ca.crt\"" Nov 03 22:08:44.932849 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.932826 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26be95\"/\"kube-root-ca.crt\"" Nov 03 22:08:44.932971 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.932940 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26be95\"/\"simulator-single-main-26be95-kserve-self-signed-certs\"" Nov 03 22:08:44.933057 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.932987 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26be95\"/\"default-dockercfg-jdzvs\"" Nov 03 22:08:44.951850 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.951826 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.951946 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.951875 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs89\" (UniqueName: \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.951946 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.951938 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.952018 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.952002 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.952093 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.952079 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:44.952128 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:44.952105 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.052875 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.052839 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.052892 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.052949 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.052980 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053087 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.053071 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053355 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.053127 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs89\" (UniqueName: \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053429 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.053378 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053998 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.053675 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.053998 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.053840 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.070989 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.070925 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.095841 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.095811 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.097152 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.097083 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs89\" (UniqueName: \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\") pod \"simulator-single-main-26be95-kserve-6644d6f5c-vh888\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.237380 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.237306 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:45.488987 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.488903 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:08:45.488987 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.488954 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 22:08:45.540842 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:45.540819 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:08:45.542720 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:08:45.542692 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a6293e_6f31_4620_b8d9_115dfdc695bd.slice/crio-e89888697c1c8335e5777f25c6215e163f7cc3c891731f1b122e03c8aeb18620 WatchSource:0}: Error finding container e89888697c1c8335e5777f25c6215e163f7cc3c891731f1b122e03c8aeb18620: Status 404 returned error can't find the container with id e89888697c1c8335e5777f25c6215e163f7cc3c891731f1b122e03c8aeb18620 Nov 03 22:08:46.088661 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.088625 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" event={"ID":"97a6293e-6f31-4620-b8d9-115dfdc695bd","Type":"ContainerStarted","Data":"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839"} Nov 03 22:08:46.088661 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.088667 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" event={"ID":"97a6293e-6f31-4620-b8d9-115dfdc695bd","Type":"ContainerStarted","Data":"e89888697c1c8335e5777f25c6215e163f7cc3c891731f1b122e03c8aeb18620"} Nov 03 22:08:46.110239 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.110193 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" podStartSLOduration=2.11017848 podStartE2EDuration="2.11017848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:46.109000718 +0000 UTC m=+1365.883850566" watchObservedRunningTime="2026-11-03 22:08:46.11017848 +0000 UTC m=+1365.885028334" Nov 03 22:08:46.238121 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.238082 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:46.481563 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.481275 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:08:46.488680 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.488648 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:08:46.488838 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:46.488707 5973 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 22:08:47.621363 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:47.621335 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:47.621929 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:47.621750 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:47.623199 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:47.623175 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:08:55.238509 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:55.238447 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:55.251281 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:55.251250 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:56.157548 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:56.157510 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:08:59.024706 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:59.024633 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j"] Nov 03 22:08:59.027053 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:08:59.027031 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-phg5j"] Nov 03 22:09:00.345371 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:00.345340 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969200da-2b26-4dd9-98af-9b4603a8b349" path="/var/lib/kubelet/pods/969200da-2b26-4dd9-98af-9b4603a8b349/volumes" Nov 03 22:09:01.458115 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:01.458085 5973 scope.go:122] "RemoveContainer" containerID="afae7a3ab57fbbb05ef3bf9ee41e6b032738d766e6c8359145b31f942d31a78d" Nov 03 22:09:01.467648 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:01.467625 5973 scope.go:122] "RemoveContainer" containerID="8b2db8dde955de1fdca1b7ff9d771a9ebcae5b70df627f8a53656f996f33b4e4" Nov 03 22:09:01.476234 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:01.476211 5973 scope.go:122] "RemoveContainer" containerID="d4d85fb266ed4b3de01b95410d8561643fe5caab482fddb7fdd01d7076325e98" Nov 03 22:09:01.484604 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:01.484580 5973 scope.go:122] "RemoveContainer" containerID="5e306dfc17922f1bee0ab4b60d02e8bc1c123213fcf7c45ec2c0657c81b4a9e2" Nov 03 22:09:02.047481 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.047431 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:09:02.358741 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.358663 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m"] Nov 03 22:09:02.364162 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.364139 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.369341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.369265 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:09:02.369341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.369284 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:09:02.369341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.369309 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:09:02.369686 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.369431 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:09:02.372614 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.372404 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m"] Nov 03 22:09:02.459792 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.459757 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.460184 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.459806 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbrg\" (UniqueName: \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.460184 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.459964 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.560902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.560864 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.560902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.560905 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbrg\" (UniqueName: \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.561173 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.560963 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.561970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.561939 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.563719 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.563690 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.574980 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.574951 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbrg\" (UniqueName: \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\") pod \"maas-api-revoke-sub-529a373e-6w74m\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.679960 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.679916 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:02.957083 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:02.957061 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m"] Nov 03 22:09:02.958234 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:09:02.958205 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f780388_390b_41bc_a296_9c4e9c13390d.slice/crio-e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a WatchSource:0}: Error finding container e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a: Status 404 returned error can't find the container with id e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a Nov 03 22:09:03.189275 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:03.189240 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" event={"ID":"4f780388-390b-41bc-a296-9c4e9c13390d","Type":"ContainerStarted","Data":"e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a"} Nov 03 22:09:04.200803 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:04.200770 5973 generic.go:356] "Generic (PLEG): container finished" podID="4f780388-390b-41bc-a296-9c4e9c13390d" containerID="687a95de580d15c8dea9c6d6fdb6378baac7c8f1c9ba7fd10eb3649a4f113d2a" exitCode=0 Nov 03 22:09:04.201358 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:04.200832 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" event={"ID":"4f780388-390b-41bc-a296-9c4e9c13390d","Type":"ContainerDied","Data":"687a95de580d15c8dea9c6d6fdb6378baac7c8f1c9ba7fd10eb3649a4f113d2a"} Nov 03 22:09:05.340162 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.340140 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:05.491432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.491354 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\") pod \"4f780388-390b-41bc-a296-9c4e9c13390d\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " Nov 03 22:09:05.491432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.491404 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\" (UniqueName: \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\") pod \"4f780388-390b-41bc-a296-9c4e9c13390d\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " Nov 03 22:09:05.491677 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.491470 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\") pod \"4f780388-390b-41bc-a296-9c4e9c13390d\" (UID: \"4f780388-390b-41bc-a296-9c4e9c13390d\") " Nov 03 22:09:05.491874 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.491850 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca" pod "4f780388-390b-41bc-a296-9c4e9c13390d" (UID: "4f780388-390b-41bc-a296-9c4e9c13390d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:05.493531 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.493490 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca" pod "4f780388-390b-41bc-a296-9c4e9c13390d" (UID: "4f780388-390b-41bc-a296-9c4e9c13390d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:05.493695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.493675 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg" pod "4f780388-390b-41bc-a296-9c4e9c13390d" (UID: "4f780388-390b-41bc-a296-9c4e9c13390d"). InnerVolumeSpecName "kube-api-access-4mbrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:05.592596 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.592557 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:05.592596 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.592589 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbrg\" (UniqueName: \"kubernetes.io/projected/4f780388-390b-41bc-a296-9c4e9c13390d-kube-api-access-4mbrg\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:05.592788 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.592603 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f780388-390b-41bc-a296-9c4e9c13390d-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:05.964765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:05.964733 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:09:06.212929 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:06.212900 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" Nov 03 22:09:06.213099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:06.212937 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m" event={"ID":"4f780388-390b-41bc-a296-9c4e9c13390d","Type":"ContainerDied","Data":"e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a"} Nov 03 22:09:06.213099 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:06.212984 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e17e18acf512674249baa782d871efc081df26ed1de5f3eb8dfe3fe00cf0313a" Nov 03 22:09:13.161919 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.161877 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc"] Nov 03 22:09:13.162345 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.162308 5973 cpu_manager.go:414] "containerMap: removing container" podUID="4f780388-390b-41bc-a296-9c4e9c13390d" containerName="revoke-keys" Nov 03 22:09:13.162345 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.162321 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f780388-390b-41bc-a296-9c4e9c13390d" containerName="revoke-keys" Nov 03 22:09:13.162485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.162391 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f780388-390b-41bc-a296-9c4e9c13390d" containerName="revoke-keys" Nov 03 22:09:13.165821 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.165771 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.172529 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.172439 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:09:13.172703 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.172639 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:09:13.172811 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.172719 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:09:13.172811 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.172782 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:09:13.174931 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.174910 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc"] Nov 03 22:09:13.264015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.263985 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.264184 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.264030 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.264184 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.264160 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rtq\" (UniqueName: \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.364855 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.364824 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rtq\" (UniqueName: \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.365019 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.364885 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.365019 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.364917 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.365677 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.365657 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.367488 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.367449 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.376888 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.376866 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rtq\" (UniqueName: \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.481077 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.481015 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:13.758634 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:13.758601 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc"] Nov 03 22:09:13.760288 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:09:13.760259 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd388251b_ddcb_4b1e_b638_9d663ea7432f.slice/crio-b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508 WatchSource:0}: Error finding container b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508: Status 404 returned error can't find the container with id b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508 Nov 03 22:09:14.261130 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:14.261087 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" event={"ID":"d388251b-ddcb-4b1e-b638-9d663ea7432f","Type":"ContainerStarted","Data":"3e7fd0407d3345c2555fdc391ca8784425afc854180719d4d55d00536cdacb19"} Nov 03 22:09:14.261580 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:14.261150 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" event={"ID":"d388251b-ddcb-4b1e-b638-9d663ea7432f","Type":"ContainerStarted","Data":"b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508"} Nov 03 22:09:14.281544 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:14.281506 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" podStartSLOduration=1.281493791 podStartE2EDuration="1.281493791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:14.279554666 +0000 UTC m=+1394.054404511" watchObservedRunningTime="2026-11-03 22:09:14.281493791 +0000 UTC m=+1394.056343669" Nov 03 22:09:14.398924 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:09:14.398878 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:09:15.269164 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:15.269124 5973 generic.go:356] "Generic (PLEG): container finished" podID="d388251b-ddcb-4b1e-b638-9d663ea7432f" containerID="3e7fd0407d3345c2555fdc391ca8784425afc854180719d4d55d00536cdacb19" exitCode=0 Nov 03 22:09:15.269564 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:15.269209 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" event={"ID":"d388251b-ddcb-4b1e-b638-9d663ea7432f","Type":"ContainerDied","Data":"3e7fd0407d3345c2555fdc391ca8784425afc854180719d4d55d00536cdacb19"} Nov 03 22:09:16.411791 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.411766 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:16.493163 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.493123 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\") pod \"d388251b-ddcb-4b1e-b638-9d663ea7432f\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " Nov 03 22:09:16.493359 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.493169 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\") pod \"d388251b-ddcb-4b1e-b638-9d663ea7432f\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " Nov 03 22:09:16.493359 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.493255 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\" (UniqueName: \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\") pod \"d388251b-ddcb-4b1e-b638-9d663ea7432f\" (UID: \"d388251b-ddcb-4b1e-b638-9d663ea7432f\") " Nov 03 22:09:16.493585 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.493560 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca" pod "d388251b-ddcb-4b1e-b638-9d663ea7432f" (UID: "d388251b-ddcb-4b1e-b638-9d663ea7432f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:16.493661 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.493644 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:16.495538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.495515 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca" pod "d388251b-ddcb-4b1e-b638-9d663ea7432f" (UID: "d388251b-ddcb-4b1e-b638-9d663ea7432f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:16.495612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.495589 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq" pod "d388251b-ddcb-4b1e-b638-9d663ea7432f" (UID: "d388251b-ddcb-4b1e-b638-9d663ea7432f"). InnerVolumeSpecName "kube-api-access-j8rtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:16.594291 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.594216 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d388251b-ddcb-4b1e-b638-9d663ea7432f-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:16.594291 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:16.594242 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rtq\" (UniqueName: \"kubernetes.io/projected/d388251b-ddcb-4b1e-b638-9d663ea7432f-kube-api-access-j8rtq\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:17.285348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:17.285310 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" event={"ID":"d388251b-ddcb-4b1e-b638-9d663ea7432f","Type":"ContainerDied","Data":"b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508"} Nov 03 22:09:17.285522 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:17.285351 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc" Nov 03 22:09:17.285522 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:17.285391 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04f62604de40b9bd9d10e37fc33f0ffe8b006be9e92498f54149c3825f9f508" Nov 03 22:09:17.995554 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:17.995524 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:09:23.348634 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:23.348598 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:09:23.350956 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:23.350930 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:09:24.688188 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:24.688150 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc"] Nov 03 22:09:24.693859 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:24.693833 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26be95-zp8nc"] Nov 03 22:09:25.840483 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:25.840438 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:09:26.344135 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:26.344103 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d388251b-ddcb-4b1e-b638-9d663ea7432f" path="/var/lib/kubelet/pods/d388251b-ddcb-4b1e-b638-9d663ea7432f/volumes" Nov 03 22:09:28.349321 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:28.349287 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" containerID="cri-o://82d81fed1c6761fc5cc04a5d3aba4124ae7065394ee314543c6eb54b548418e4" gracePeriod=25 Nov 03 22:09:28.351704 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:28.351662 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" containerID="cri-o://c337128a93a981fd1bd801c0ff7c82b4c5f726f619c33f3f3a7a00cacf6eef3e" gracePeriod=25 Nov 03 22:09:31.656836 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:31.656799 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:09:31.657251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:31.657062 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" containerID="cri-o://228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f" gracePeriod=30 Nov 03 22:09:31.952615 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:31.952528 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:09:31.952615 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:31.952576 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:09:32.624437 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:32.624393 5973 patch_prober.go:29] interesting pod/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:09:32.624710 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:32.624478 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:09:33.016355 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.016298 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerName="main" containerID="cri-o://61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839" gracePeriod=45 Nov 03 22:09:33.147436 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.147395 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": dial tcp 10.134.0.111:8000: connect: connection refused" Nov 03 22:09:33.269690 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.269633 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:09:33.345057 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345022 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345107 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345143 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345164 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345223 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\" (UniqueName: \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345260 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\") pod \"97a6293e-6f31-4620-b8d9-115dfdc695bd\" (UID: \"97a6293e-6f31-4620-b8d9-115dfdc695bd\") " Nov 03 22:09:33.345500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345399 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:33.345500 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345441 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:33.345688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345594 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.345688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345607 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.345688 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.345625 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:33.347605 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.347577 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "kube-api-access-7qs89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:33.347863 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.347838 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:33.347981 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.347860 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs" pod "97a6293e-6f31-4620-b8d9-115dfdc695bd" (UID: "97a6293e-6f31-4620-b8d9-115dfdc695bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:33.374341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.374318 5973 generic.go:356] "Generic (PLEG): container finished" podID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerID="61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839" exitCode=0 Nov 03 22:09:33.374435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.374362 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" Nov 03 22:09:33.374435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.374398 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" event={"ID":"97a6293e-6f31-4620-b8d9-115dfdc695bd","Type":"ContainerDied","Data":"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839"} Nov 03 22:09:33.374539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.374440 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888" event={"ID":"97a6293e-6f31-4620-b8d9-115dfdc695bd","Type":"ContainerDied","Data":"e89888697c1c8335e5777f25c6215e163f7cc3c891731f1b122e03c8aeb18620"} Nov 03 22:09:33.374539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.374473 5973 scope.go:122] "RemoveContainer" containerID="61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839" Nov 03 22:09:33.384187 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.384173 5973 scope.go:122] "RemoveContainer" containerID="61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839" Nov 03 22:09:33.384494 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:09:33.384446 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839\": container with ID starting with 61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839 not found: ID does not exist" containerID="61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839" Nov 03 22:09:33.384554 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.384503 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839"} err="failed to get container status \"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839\": rpc error: code = NotFound desc = could not find container \"61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839\": container with ID starting with 61ac33db475c875ec25535066cbce6a4bd0dee452618c1df8db737f152395839 not found: ID does not exist" Nov 03 22:09:33.396652 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.396620 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:09:33.400528 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.400507 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-26be95/simulator-single-main-26be95-kserve-6644d6f5c-vh888"] Nov 03 22:09:33.446841 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.446820 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qs89\" (UniqueName: \"kubernetes.io/projected/97a6293e-6f31-4620-b8d9-115dfdc695bd-kube-api-access-7qs89\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.446961 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.446843 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.446961 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.446854 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a6293e-6f31-4620-b8d9-115dfdc695bd-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:33.446961 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:33.446863 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a6293e-6f31-4620-b8d9-115dfdc695bd-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:34.346186 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:34.346154 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" path="/var/lib/kubelet/pods/97a6293e-6f31-4620-b8d9-115dfdc695bd/volumes" Nov 03 22:09:35.427859 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.427826 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:09:35.428419 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428400 5973 cpu_manager.go:414] "containerMap: removing container" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerName="main" Nov 03 22:09:35.428523 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428422 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerName="main" Nov 03 22:09:35.428523 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428447 5973 cpu_manager.go:414] "containerMap: removing container" podUID="d388251b-ddcb-4b1e-b638-9d663ea7432f" containerName="revoke-keys" Nov 03 22:09:35.428523 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428472 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d388251b-ddcb-4b1e-b638-9d663ea7432f" containerName="revoke-keys" Nov 03 22:09:35.428737 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428578 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="d388251b-ddcb-4b1e-b638-9d663ea7432f" containerName="revoke-keys" Nov 03 22:09:35.428737 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.428598 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a6293e-6f31-4620-b8d9-115dfdc695bd" containerName="main" Nov 03 22:09:35.432916 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.432895 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.441643 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.441619 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e26ab648-kserve-self-signed-certs\"" Nov 03 22:09:35.441643 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.441631 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:09:35.567677 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567643 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.567862 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567681 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.567862 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567791 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.567862 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567823 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.568001 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567863 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j86\" (UniqueName: \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.568001 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.567909 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669108 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669154 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669191 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669211 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669232 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j86\" (UniqueName: \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669384 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669249 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669633 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669603 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669633 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669615 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.669711 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.669666 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.671670 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.671644 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.671949 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.671929 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.681241 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.681186 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j86\" (UniqueName: \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\") pod \"e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:35.745721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:35.745693 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:36.023840 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.023817 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:09:36.025149 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:09:36.025120 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda867ad1b_fa59_4c4f_941c_e4b56aa39281.slice/crio-72a770c099ef60c3e266e78ffb79a51f9da4f4e6e49efbd7b50bcc56218788b3 WatchSource:0}: Error finding container 72a770c099ef60c3e266e78ffb79a51f9da4f4e6e49efbd7b50bcc56218788b3: Status 404 returned error can't find the container with id 72a770c099ef60c3e266e78ffb79a51f9da4f4e6e49efbd7b50bcc56218788b3 Nov 03 22:09:36.393053 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.393015 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" event={"ID":"a867ad1b-fa59-4c4f-941c-e4b56aa39281","Type":"ContainerStarted","Data":"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a"} Nov 03 22:09:36.393227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.393064 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" event={"ID":"a867ad1b-fa59-4c4f-941c-e4b56aa39281","Type":"ContainerStarted","Data":"72a770c099ef60c3e266e78ffb79a51f9da4f4e6e49efbd7b50bcc56218788b3"} Nov 03 22:09:36.413912 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.413867 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" podStartSLOduration=1.413852156 podStartE2EDuration="1.413852156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:36.411310064 +0000 UTC m=+1416.186159914" watchObservedRunningTime="2026-11-03 22:09:36.413852156 +0000 UTC m=+1416.188702003" Nov 03 22:09:36.746415 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.746388 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:36.905914 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.905888 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:09:36.982344 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982260 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982344 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982317 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982393 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982447 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982497 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982530 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982574 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982587 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982569 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:36.982891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982600 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982665 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\") pod \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\" (UID: \"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5\") " Nov 03 22:09:36.982891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982711 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:36.983038 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982952 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:36.983038 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.982979 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:36.983139 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.983076 5973 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:36.983139 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.983094 5973 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-data\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:36.983139 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.983108 5973 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-credential-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:36.983139 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.983122 5973 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istiod-ca-cert\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:36.983523 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.983496 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:36.984873 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.984849 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:36.985262 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.985239 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:09:36.985366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.985326 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:36.985366 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:36.985343 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl" pod "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" (UID: "6f0077f5-7c4e-44fe-b2bc-59b6ab837df5"). InnerVolumeSpecName "kube-api-access-qzmvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:37.022822 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.022774 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:09:37.084720 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.084686 5973 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-envoy\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:37.084720 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.084719 5973 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-token\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:37.084958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.084731 5973 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-workload-socket\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:37.084958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.084745 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzmvl\" (UniqueName: \"kubernetes.io/projected/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-kube-api-access-qzmvl\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:37.084958 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.084754 5973 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5-istio-podinfo\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:37.402181 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.402150 5973 generic.go:356] "Generic (PLEG): container finished" podID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerID="228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f" exitCode=0 Nov 03 22:09:37.402348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.402222 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" Nov 03 22:09:37.402348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.402242 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerDied","Data":"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f"} Nov 03 22:09:37.402348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.402307 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m" event={"ID":"6f0077f5-7c4e-44fe-b2bc-59b6ab837df5","Type":"ContainerDied","Data":"13faf447e280e43aa7775c6d14027ca5443058d6f4cf3041c750fd0011dd7e76"} Nov 03 22:09:37.402348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.402337 5973 scope.go:122] "RemoveContainer" containerID="228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f" Nov 03 22:09:37.412077 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.412061 5973 scope.go:122] "RemoveContainer" containerID="3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908" Nov 03 22:09:37.421864 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.421845 5973 scope.go:122] "RemoveContainer" containerID="228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f" Nov 03 22:09:37.422136 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:09:37.422115 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f\": container with ID starting with 228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f not found: ID does not exist" containerID="228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f" Nov 03 22:09:37.422198 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.422144 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f"} err="failed to get container status \"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f\": rpc error: code = NotFound desc = could not find container \"228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f\": container with ID starting with 228a1f2f1e95d49469a6c0649fc11bf649adc01f65332b00113345bbc9bf539f not found: ID does not exist" Nov 03 22:09:37.422198 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.422159 5973 scope.go:122] "RemoveContainer" containerID="3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908" Nov 03 22:09:37.422429 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:09:37.422414 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908\": container with ID starting with 3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908 not found: ID does not exist" containerID="3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908" Nov 03 22:09:37.422534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.422434 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908"} err="failed to get container status \"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908\": rpc error: code = NotFound desc = could not find container \"3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908\": container with ID starting with 3c67ccf25c87f1cdaa3e4415bbcdd7218147a9de24ea37144fe8452d08ba5908 not found: ID does not exist" Nov 03 22:09:37.429058 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.429037 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:09:37.432253 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:37.432233 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26be95-openshift-default-666746c49csdr4m"] Nov 03 22:09:38.347058 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:38.347021 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" path="/var/lib/kubelet/pods/6f0077f5-7c4e-44fe-b2bc-59b6ab837df5/volumes" Nov 03 22:09:41.952943 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:41.952908 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:09:41.952943 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:41.952924 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:09:44.399484 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:09:44.399428 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:09:45.746040 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:45.746005 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:45.758012 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:45.757993 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:46.483663 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:46.483629 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:09:48.487892 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.487858 5973 generic.go:356] "Generic (PLEG): container finished" podID="102f3380-0727-4b3e-8c76-20eafa08c763" containerID="82d81fed1c6761fc5cc04a5d3aba4124ae7065394ee314543c6eb54b548418e4" exitCode=0 Nov 03 22:09:48.488327 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.487930 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" event={"ID":"102f3380-0727-4b3e-8c76-20eafa08c763","Type":"ContainerDied","Data":"82d81fed1c6761fc5cc04a5d3aba4124ae7065394ee314543c6eb54b548418e4"} Nov 03 22:09:48.489697 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.489675 5973 generic.go:356] "Generic (PLEG): container finished" podID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerID="c337128a93a981fd1bd801c0ff7c82b4c5f726f619c33f3f3a7a00cacf6eef3e" exitCode=0 Nov 03 22:09:48.489813 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.489731 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" event={"ID":"3acb6a93-f1aa-42dc-b745-60f834a89e2c","Type":"ContainerDied","Data":"c337128a93a981fd1bd801c0ff7c82b4c5f726f619c33f3f3a7a00cacf6eef3e"} Nov 03 22:09:48.611149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.611091 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:09:48.638521 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.638500 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:09:48.712785 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.712754 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\") pod \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " Nov 03 22:09:48.712954 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.712792 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\") pod \"102f3380-0727-4b3e-8c76-20eafa08c763\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " Nov 03 22:09:48.712954 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.712830 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\" (UniqueName: \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\") pod \"102f3380-0727-4b3e-8c76-20eafa08c763\" (UID: \"102f3380-0727-4b3e-8c76-20eafa08c763\") " Nov 03 22:09:48.712954 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.712902 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\" (UniqueName: \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\") pod \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\" (UID: \"3acb6a93-f1aa-42dc-b745-60f834a89e2c\") " Nov 03 22:09:48.713213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.713189 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume" pod "3acb6a93-f1aa-42dc-b745-60f834a89e2c" (UID: "3acb6a93-f1aa-42dc-b745-60f834a89e2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:48.713269 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.713208 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume" pod "102f3380-0727-4b3e-8c76-20eafa08c763" (UID: "102f3380-0727-4b3e-8c76-20eafa08c763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:48.715154 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.715133 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr" pod "3acb6a93-f1aa-42dc-b745-60f834a89e2c" (UID: "3acb6a93-f1aa-42dc-b745-60f834a89e2c"). InnerVolumeSpecName "kube-api-access-xvjcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:48.715223 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.715139 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn" pod "102f3380-0727-4b3e-8c76-20eafa08c763" (UID: "102f3380-0727-4b3e-8c76-20eafa08c763"). InnerVolumeSpecName "kube-api-access-5cnqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:48.814474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.814432 5973 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acb6a93-f1aa-42dc-b745-60f834a89e2c-plugins-config-volume\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:48.814474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.814452 5973 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102f3380-0727-4b3e-8c76-20eafa08c763-plugins-config-volume\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:48.814474 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.814475 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnqn\" (UniqueName: \"kubernetes.io/projected/102f3380-0727-4b3e-8c76-20eafa08c763-kube-api-access-5cnqn\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:48.814659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:48.814484 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjcr\" (UniqueName: \"kubernetes.io/projected/3acb6a93-f1aa-42dc-b745-60f834a89e2c-kube-api-access-xvjcr\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:49.498176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.498143 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" Nov 03 22:09:49.498176 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.498171 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx" event={"ID":"102f3380-0727-4b3e-8c76-20eafa08c763","Type":"ContainerDied","Data":"2264de87d14ec72fe6b998ac90c2f1a6537e49a328bacf7964bd1a3efc46839c"} Nov 03 22:09:49.498770 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.498229 5973 scope.go:122] "RemoveContainer" containerID="82d81fed1c6761fc5cc04a5d3aba4124ae7065394ee314543c6eb54b548418e4" Nov 03 22:09:49.500041 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.500010 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" event={"ID":"3acb6a93-f1aa-42dc-b745-60f834a89e2c","Type":"ContainerDied","Data":"d4abeff8efe5e53afa15fff8e95024d78787abd22185a47ced01954372aa8528"} Nov 03 22:09:49.500158 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.500021 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9" Nov 03 22:09:49.508023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.508008 5973 scope.go:122] "RemoveContainer" containerID="c337128a93a981fd1bd801c0ff7c82b4c5f726f619c33f3f3a7a00cacf6eef3e" Nov 03 22:09:49.523690 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.523670 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:09:49.528147 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.528127 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26be95-f88f94l9m9"] Nov 03 22:09:49.539232 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.539209 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:09:49.543321 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:49.543300 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26be95-59478f985bnrcx"] Nov 03 22:09:50.344661 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:50.344632 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" path="/var/lib/kubelet/pods/102f3380-0727-4b3e-8c76-20eafa08c763/volumes" Nov 03 22:09:50.344995 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:50.344983 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" path="/var/lib/kubelet/pods/3acb6a93-f1aa-42dc-b745-60f834a89e2c/volumes" Nov 03 22:09:51.980260 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.980223 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:09:51.980983 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.980961 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.981035 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.980990 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.981035 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981009 5973 cpu_manager.go:414] "containerMap: removing container" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.981035 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981017 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.981149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981049 5973 cpu_manager.go:414] "containerMap: removing container" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" Nov 03 22:09:51.981149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981059 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" Nov 03 22:09:51.981149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981100 5973 cpu_manager.go:414] "containerMap: removing container" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" Nov 03 22:09:51.981149 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981109 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" Nov 03 22:09:51.981285 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981246 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f3380-0727-4b3e-8c76-20eafa08c763" containerName="payload-processing" Nov 03 22:09:51.981285 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981267 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acb6a93-f1aa-42dc-b745-60f834a89e2c" containerName="payload-pre-processing" Nov 03 22:09:51.981285 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981280 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.981380 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.981289 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0077f5-7c4e-44fe-b2bc-59b6ab837df5" containerName="istio-proxy" Nov 03 22:09:51.986650 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.986627 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:51.992504 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.992473 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e26ab648-kserve-self-signed-certs\"" Nov 03 22:09:51.994245 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:51.994206 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:09:52.045468 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045426 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.045695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045492 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.045695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045615 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.045695 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045651 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8tt7\" (UniqueName: \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.045809 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045779 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.045854 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.045841 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147268 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147232 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147280 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147299 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147355 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147377 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8tt7\" (UniqueName: \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147440 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147752 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147892 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.147978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.147925 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.150150 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.150128 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.150428 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.150403 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.160179 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.160152 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8tt7\" (UniqueName: \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\") pod \"e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.300723 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.300615 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:09:52.580515 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:52.580491 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:09:52.582244 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:09:52.582218 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba1c559e_7520_4ddb_a1ad_2a75bdab5eac.slice/crio-d083824e837ac9024e1b54a845778c41c489014b1596884410fc4fdd63f0420a WatchSource:0}: Error finding container d083824e837ac9024e1b54a845778c41c489014b1596884410fc4fdd63f0420a: Status 404 returned error can't find the container with id d083824e837ac9024e1b54a845778c41c489014b1596884410fc4fdd63f0420a Nov 03 22:09:53.466752 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:53.466717 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:09:53.526982 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:53.526940 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" event={"ID":"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac","Type":"ContainerStarted","Data":"89d2f4164a58753629291856f0583ecc945f06d6f54842ffc2db51501518e2ba"} Nov 03 22:09:53.527143 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:53.526990 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" event={"ID":"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac","Type":"ContainerStarted","Data":"d083824e837ac9024e1b54a845778c41c489014b1596884410fc4fdd63f0420a"} Nov 03 22:09:53.574177 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:53.574130 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" podStartSLOduration=2.574117627 podStartE2EDuration="2.574117627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:53.57137362 +0000 UTC m=+1433.346223467" watchObservedRunningTime="2026-11-03 22:09:53.574117627 +0000 UTC m=+1433.348967474" Nov 03 22:09:54.302254 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:09:54.301520 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:01.524006 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:01.523972 5973 scope.go:122] "RemoveContainer" containerID="207898bb81e5a0714a6a9812dcd9795f79d18e94f7840aac3ea561eb687bb2b4" Nov 03 22:10:01.533067 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:01.533050 5973 scope.go:122] "RemoveContainer" containerID="e4478c92c4c1efbd3cfde3bc431513f4949559558fb375d312959767c64afd85" Nov 03 22:10:01.541349 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:01.541330 5973 scope.go:122] "RemoveContainer" containerID="b1323584cc5db8044e5b58839057be7ed99a8eea052862454cdf7574cb17221f" Nov 03 22:10:02.300955 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:02.300913 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:02.313575 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:02.313547 5973 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:02.592685 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:02.592618 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:09.408218 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.408182 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd"] Nov 03 22:10:09.413375 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.413355 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.421360 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.421335 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:10:09.421507 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.421376 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:10:09.421507 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.421401 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:10:09.421507 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.421444 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:10:09.423760 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.423742 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd"] Nov 03 22:10:09.455970 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.455937 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.456133 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.456006 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.456133 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.456093 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224g9\" (UniqueName: \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.556634 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.556605 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.556828 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.556652 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.556828 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.556682 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224g9\" (UniqueName: \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.557748 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.557714 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.560936 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.560906 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.569758 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.569734 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-224g9\" (UniqueName: \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\") pod \"maas-api-revoke-sub-452b6d7b-8ddhd\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.727283 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.727197 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:09.771685 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:09.771651 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:10:10.003946 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:10.003915 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd"] Nov 03 22:10:10.005317 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:10:10.005287 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a4f860_3c4a_4ce7_8357_542380fa1705.slice/crio-98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7 WatchSource:0}: Error finding container 98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7: Status 404 returned error can't find the container with id 98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7 Nov 03 22:10:10.629211 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:10.629103 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" event={"ID":"08a4f860-3c4a-4ce7-8357-542380fa1705","Type":"ContainerStarted","Data":"9346b52c28bd24ad432b6f04e7e6bc60b7dfe41a85829d0d1055f87c5e745479"} Nov 03 22:10:10.629211 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:10.629190 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" event={"ID":"08a4f860-3c4a-4ce7-8357-542380fa1705","Type":"ContainerStarted","Data":"98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7"} Nov 03 22:10:10.652167 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:10.652110 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" podStartSLOduration=1.652091924 podStartE2EDuration="1.652091924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:10.648920338 +0000 UTC m=+1450.423770185" watchObservedRunningTime="2026-11-03 22:10:10.652091924 +0000 UTC m=+1450.426941772" Nov 03 22:10:11.638250 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:11.638209 5973 generic.go:356] "Generic (PLEG): container finished" podID="08a4f860-3c4a-4ce7-8357-542380fa1705" containerID="9346b52c28bd24ad432b6f04e7e6bc60b7dfe41a85829d0d1055f87c5e745479" exitCode=0 Nov 03 22:10:11.638250 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:11.638247 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" event={"ID":"08a4f860-3c4a-4ce7-8357-542380fa1705","Type":"ContainerDied","Data":"9346b52c28bd24ad432b6f04e7e6bc60b7dfe41a85829d0d1055f87c5e745479"} Nov 03 22:10:12.985483 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:12.980655 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:13.139480 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.139091 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\") pod \"08a4f860-3c4a-4ce7-8357-542380fa1705\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " Nov 03 22:10:13.139480 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.139164 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\" (UniqueName: \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\") pod \"08a4f860-3c4a-4ce7-8357-542380fa1705\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " Nov 03 22:10:13.139480 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.139209 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\") pod \"08a4f860-3c4a-4ce7-8357-542380fa1705\" (UID: \"08a4f860-3c4a-4ce7-8357-542380fa1705\") " Nov 03 22:10:13.140494 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.140069 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca" pod "08a4f860-3c4a-4ce7-8357-542380fa1705" (UID: "08a4f860-3c4a-4ce7-8357-542380fa1705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:10:13.142334 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.142311 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9" pod "08a4f860-3c4a-4ce7-8357-542380fa1705" (UID: "08a4f860-3c4a-4ce7-8357-542380fa1705"). InnerVolumeSpecName "kube-api-access-224g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:13.142484 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.142413 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca" pod "08a4f860-3c4a-4ce7-8357-542380fa1705" (UID: "08a4f860-3c4a-4ce7-8357-542380fa1705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:13.240178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.240084 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-224g9\" (UniqueName: \"kubernetes.io/projected/08a4f860-3c4a-4ce7-8357-542380fa1705-kube-api-access-224g9\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:13.240178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.240117 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:13.240178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.240126 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4f860-3c4a-4ce7-8357-542380fa1705-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:13.650276 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.650244 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" Nov 03 22:10:13.650480 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.650275 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8ddhd" event={"ID":"08a4f860-3c4a-4ce7-8357-542380fa1705","Type":"ContainerDied","Data":"98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7"} Nov 03 22:10:13.650480 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:13.650325 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d4ee7a6db87fd5450424e768a2da4995d9300058f90487502a5a51cf53d6a7" Nov 03 22:10:14.399905 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:10:14.399869 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:10:20.255556 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:20.255520 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:10:21.524335 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:21.524298 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:10:22.465306 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:22.465272 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:10:25.711026 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.710987 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl"] Nov 03 22:10:25.711660 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.711616 5973 cpu_manager.go:414] "containerMap: removing container" podUID="08a4f860-3c4a-4ce7-8357-542380fa1705" containerName="revoke-keys" Nov 03 22:10:25.711660 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.711635 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a4f860-3c4a-4ce7-8357-542380fa1705" containerName="revoke-keys" Nov 03 22:10:25.711787 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.711762 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a4f860-3c4a-4ce7-8357-542380fa1705" containerName="revoke-keys" Nov 03 22:10:25.714731 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.714710 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.725340 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.724542 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:10:25.725340 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.725217 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:10:25.728659 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.728480 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl"] Nov 03 22:10:25.732734 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.732711 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:10:25.732898 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.732724 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:10:25.758227 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.758172 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.758427 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.758411 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qr5\" (UniqueName: \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.758548 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.758529 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.860051 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.860004 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.860237 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.860079 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qr5\" (UniqueName: \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.860237 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.860142 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.860925 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.860889 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.862766 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.862739 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:25.873088 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:25.873061 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qr5\" (UniqueName: \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\") pod \"maas-api-revoke-sub-3e31a506-76xdl\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:26.032507 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:26.032402 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:26.346229 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:10:26.346202 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166b46e6_7247_467e_80fd_652e03ec045b.slice/crio-380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3 WatchSource:0}: Error finding container 380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3: Status 404 returned error can't find the container with id 380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3 Nov 03 22:10:26.349358 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:26.349334 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl"] Nov 03 22:10:26.769479 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:26.765733 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" event={"ID":"166b46e6-7247-467e-80fd-652e03ec045b","Type":"ContainerStarted","Data":"aa682ef8cafe2069e4ab371021865ef7eeb3b88355839994c14567959a80697c"} Nov 03 22:10:26.769479 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:26.765800 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" event={"ID":"166b46e6-7247-467e-80fd-652e03ec045b","Type":"ContainerStarted","Data":"380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3"} Nov 03 22:10:27.802376 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:27.802286 5973 generic.go:356] "Generic (PLEG): container finished" podID="166b46e6-7247-467e-80fd-652e03ec045b" containerID="aa682ef8cafe2069e4ab371021865ef7eeb3b88355839994c14567959a80697c" exitCode=0 Nov 03 22:10:27.803620 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:27.803589 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" event={"ID":"166b46e6-7247-467e-80fd-652e03ec045b","Type":"ContainerDied","Data":"aa682ef8cafe2069e4ab371021865ef7eeb3b88355839994c14567959a80697c"} Nov 03 22:10:28.373549 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:28.373519 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:10:28.942971 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:28.942940 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:29.050283 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.050256 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\") pod \"166b46e6-7247-467e-80fd-652e03ec045b\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " Nov 03 22:10:29.050486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.050289 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\" (UniqueName: \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\") pod \"166b46e6-7247-467e-80fd-652e03ec045b\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " Nov 03 22:10:29.050486 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.050313 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\") pod \"166b46e6-7247-467e-80fd-652e03ec045b\" (UID: \"166b46e6-7247-467e-80fd-652e03ec045b\") " Nov 03 22:10:29.050713 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.050690 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca" pod "166b46e6-7247-467e-80fd-652e03ec045b" (UID: "166b46e6-7247-467e-80fd-652e03ec045b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:10:29.052487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.052451 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5" pod "166b46e6-7247-467e-80fd-652e03ec045b" (UID: "166b46e6-7247-467e-80fd-652e03ec045b"). InnerVolumeSpecName "kube-api-access-v4qr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:29.052487 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.052452 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca" pod "166b46e6-7247-467e-80fd-652e03ec045b" (UID: "166b46e6-7247-467e-80fd-652e03ec045b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:29.151448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.151418 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:29.151448 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.151446 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qr5\" (UniqueName: \"kubernetes.io/projected/166b46e6-7247-467e-80fd-652e03ec045b-kube-api-access-v4qr5\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:29.151650 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.151487 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166b46e6-7247-467e-80fd-652e03ec045b-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:29.814637 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.814604 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" Nov 03 22:10:29.814848 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.814607 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-76xdl" event={"ID":"166b46e6-7247-467e-80fd-652e03ec045b","Type":"ContainerDied","Data":"380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3"} Nov 03 22:10:29.814848 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:29.814749 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380fd80fe90084fc62fd165732591ed6ed60a9a8e55fab942f230fb4044a56c3" Nov 03 22:10:35.276925 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.276860 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerName="main" containerID="cri-o://b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a" gracePeriod=45 Nov 03 22:10:35.532052 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.531993 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:10:35.608076 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608048 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608106 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608162 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\" (UniqueName: \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608213 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608203 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608259 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608300 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\" (UniqueName: \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\") pod \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\" (UID: \"a867ad1b-fa59-4c4f-941c-e4b56aa39281\") " Nov 03 22:10:35.608348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608320 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:35.608476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608364 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:35.608598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608576 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:35.608668 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608654 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.608716 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608670 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.608716 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.608679 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.610601 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.610578 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "kube-api-access-s7j86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:35.610721 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.610635 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:35.610765 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.610753 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm" pod "a867ad1b-fa59-4c4f-941c-e4b56aa39281" (UID: "a867ad1b-fa59-4c4f-941c-e4b56aa39281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:35.709834 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.709803 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7j86\" (UniqueName: \"kubernetes.io/projected/a867ad1b-fa59-4c4f-941c-e4b56aa39281-kube-api-access-s7j86\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.709834 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.709831 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a867ad1b-fa59-4c4f-941c-e4b56aa39281-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.709834 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.709840 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a867ad1b-fa59-4c4f-941c-e4b56aa39281-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:35.859401 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.859317 5973 generic.go:356] "Generic (PLEG): container finished" podID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerID="b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a" exitCode=0 Nov 03 22:10:35.859401 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.859358 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" Nov 03 22:10:35.859598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.859409 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" event={"ID":"a867ad1b-fa59-4c4f-941c-e4b56aa39281","Type":"ContainerDied","Data":"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a"} Nov 03 22:10:35.859598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.859451 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" event={"ID":"a867ad1b-fa59-4c4f-941c-e4b56aa39281","Type":"ContainerDied","Data":"72a770c099ef60c3e266e78ffb79a51f9da4f4e6e49efbd7b50bcc56218788b3"} Nov 03 22:10:35.859598 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.859489 5973 scope.go:122] "RemoveContainer" containerID="b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a" Nov 03 22:10:35.869413 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.869399 5973 scope.go:122] "RemoveContainer" containerID="b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a" Nov 03 22:10:35.869707 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:10:35.869689 5973 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a\": container with ID starting with b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a not found: ID does not exist" containerID="b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a" Nov 03 22:10:35.869763 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.869716 5973 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a"} err="failed to get container status \"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a\": rpc error: code = NotFound desc = could not find container \"b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a\": container with ID starting with b5f1c071423fbc1e492d03c29a29e503d6f6fd99b25eef1df88518fbc110947a not found: ID does not exist" Nov 03 22:10:35.881525 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.881506 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:10:35.887719 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:35.887700 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp"] Nov 03 22:10:36.344990 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:36.344957 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" path="/var/lib/kubelet/pods/a867ad1b-fa59-4c4f-941c-e4b56aa39281/volumes" Nov 03 22:10:36.474379 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:36.474343 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-e26ab648-kserve-cddccb78f-4xwcp" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded" Nov 03 22:10:36.538038 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:36.538006 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:10:37.504485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:37.503708 5973 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerName="main" containerID="cri-o://89d2f4164a58753629291856f0583ecc945f06d6f54842ffc2db51501518e2ba" gracePeriod=45 Nov 03 22:10:37.588913 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:37.588873 5973 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": dial tcp 10.134.0.115:8000: connect: connection refused" Nov 03 22:10:37.930991 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:37.930953 5973 generic.go:356] "Generic (PLEG): container finished" podID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerID="89d2f4164a58753629291856f0583ecc945f06d6f54842ffc2db51501518e2ba" exitCode=0 Nov 03 22:10:37.931244 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:37.931218 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" event={"ID":"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac","Type":"ContainerDied","Data":"89d2f4164a58753629291856f0583ecc945f06d6f54842ffc2db51501518e2ba"} Nov 03 22:10:38.043538 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.043514 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:38.140684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.140650 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.140981 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.140943 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.141222 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.141205 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\" (UniqueName: \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.141411 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.141380 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.141657 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.141630 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.141846 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.141815 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\") pod \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\" (UID: \"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac\") " Nov 03 22:10:38.143032 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.142983 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:38.143311 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.143291 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:38.143499 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.143480 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:38.145383 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.145355 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "kube-api-access-h8tt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:38.145540 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.145515 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:10:38.146203 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.146179 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs" pod "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" (UID: "ba1c559e-7520-4ddb-a1ad-2a75bdab5eac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:38.243201 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243166 5973 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tls-certs\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.243201 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243190 5973 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-home\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.243201 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243199 5973 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-tmp-dir\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.243201 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243207 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8tt7\" (UniqueName: \"kubernetes.io/projected/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-kube-api-access-h8tt7\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.243498 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243215 5973 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-dshm\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.243498 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.243224 5973 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac-model-cache\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:38.938113 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.938080 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" Nov 03 22:10:38.938113 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.938105 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn" event={"ID":"ba1c559e-7520-4ddb-a1ad-2a75bdab5eac","Type":"ContainerDied","Data":"d083824e837ac9024e1b54a845778c41c489014b1596884410fc4fdd63f0420a"} Nov 03 22:10:38.938667 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.938156 5973 scope.go:122] "RemoveContainer" containerID="89d2f4164a58753629291856f0583ecc945f06d6f54842ffc2db51501518e2ba" Nov 03 22:10:38.956080 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.956059 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:10:38.960935 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:38.960914 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e26ab648-kserve-5bf886d98f-gggnn"] Nov 03 22:10:40.386501 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:40.383214 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" path="/var/lib/kubelet/pods/ba1c559e-7520-4ddb-a1ad-2a75bdab5eac/volumes" Nov 03 22:10:41.027059 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:41.027011 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:10:44.398868 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:10:44.398830 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:10:46.203051 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203012 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb"] Nov 03 22:10:46.203569 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203551 5973 cpu_manager.go:414] "containerMap: removing container" podUID="166b46e6-7247-467e-80fd-652e03ec045b" containerName="revoke-keys" Nov 03 22:10:46.203623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203570 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166b46e6-7247-467e-80fd-652e03ec045b" containerName="revoke-keys" Nov 03 22:10:46.203623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203582 5973 cpu_manager.go:414] "containerMap: removing container" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerName="main" Nov 03 22:10:46.203623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203587 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerName="main" Nov 03 22:10:46.203623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203603 5973 cpu_manager.go:414] "containerMap: removing container" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerName="main" Nov 03 22:10:46.203623 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203608 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerName="main" Nov 03 22:10:46.203797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203673 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="a867ad1b-fa59-4c4f-941c-e4b56aa39281" containerName="main" Nov 03 22:10:46.203797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203685 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="166b46e6-7247-467e-80fd-652e03ec045b" containerName="revoke-keys" Nov 03 22:10:46.203797 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.203692 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1c559e-7520-4ddb-a1ad-2a75bdab5eac" containerName="main" Nov 03 22:10:46.208179 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.208152 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.214526 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.214487 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:10:46.214692 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.214547 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:10:46.217450 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.217419 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb"] Nov 03 22:10:46.263420 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.263396 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4mx\" (UniqueName: \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.263592 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.263447 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.263592 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.263567 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.268178 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.268153 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:10:46.268296 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.268212 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:10:46.364116 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.364094 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.364265 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.364156 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4mx\" (UniqueName: \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.364265 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.364198 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.364810 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.364792 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.366610 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.366590 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.376793 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.376770 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4mx\" (UniqueName: \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\") pod \"maas-api-revoke-sub-ee5f23f9-l96tb\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.524612 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.524540 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:46.801109 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.801086 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb"] Nov 03 22:10:46.802777 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:10:46.802750 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39be2535_24b7_49ee_b6ed_913b2e6c1c7a.slice/crio-70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6 WatchSource:0}: Error finding container 70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6: Status 404 returned error can't find the container with id 70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6 Nov 03 22:10:46.986263 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:46.986231 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" event={"ID":"39be2535-24b7-49ee-b6ed-913b2e6c1c7a","Type":"ContainerStarted","Data":"70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6"} Nov 03 22:10:47.993826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:47.993785 5973 generic.go:356] "Generic (PLEG): container finished" podID="39be2535-24b7-49ee-b6ed-913b2e6c1c7a" containerID="6eb4d90b269abf7e110d157010a1f7223b6055c8fb50b5eb9f4a99a3f35f11f7" exitCode=0 Nov 03 22:10:47.994282 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:47.993878 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" event={"ID":"39be2535-24b7-49ee-b6ed-913b2e6c1c7a","Type":"ContainerDied","Data":"6eb4d90b269abf7e110d157010a1f7223b6055c8fb50b5eb9f4a99a3f35f11f7"} Nov 03 22:10:48.022988 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:48.022957 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g"] Nov 03 22:10:48.026421 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:48.026393 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3900c8ee-hnv9g"] Nov 03 22:10:48.344981 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:48.344888 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e3eebe-70b1-4957-913a-ea117211d3a9" path="/var/lib/kubelet/pods/95e3eebe-70b1-4957-913a-ea117211d3a9/volumes" Nov 03 22:10:49.137997 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.137975 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:49.193215 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.193190 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\") pod \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " Nov 03 22:10:49.193348 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.193327 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\") pod \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " Nov 03 22:10:49.193411 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.193390 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\" (UniqueName: \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\") pod \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\" (UID: \"39be2535-24b7-49ee-b6ed-913b2e6c1c7a\") " Nov 03 22:10:49.193792 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.193763 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca" pod "39be2535-24b7-49ee-b6ed-913b2e6c1c7a" (UID: "39be2535-24b7-49ee-b6ed-913b2e6c1c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:10:49.195425 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.195403 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca" pod "39be2535-24b7-49ee-b6ed-913b2e6c1c7a" (UID: "39be2535-24b7-49ee-b6ed-913b2e6c1c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:10:49.195542 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.195526 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx" pod "39be2535-24b7-49ee-b6ed-913b2e6c1c7a" (UID: "39be2535-24b7-49ee-b6ed-913b2e6c1c7a"). InnerVolumeSpecName "kube-api-access-2c4mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:10:49.294769 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.294707 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:49.294769 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.294732 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4mx\" (UniqueName: \"kubernetes.io/projected/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-kube-api-access-2c4mx\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:49.294769 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:49.294742 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39be2535-24b7-49ee-b6ed-913b2e6c1c7a-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:10:50.006414 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:50.006380 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" event={"ID":"39be2535-24b7-49ee-b6ed-913b2e6c1c7a","Type":"ContainerDied","Data":"70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6"} Nov 03 22:10:50.006414 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:50.006404 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f23f9-l96tb" Nov 03 22:10:50.006643 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:50.006427 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a1e2406f8161bbd796fc5d5d6ecabd3bbceb3978991a49a7e800dcb28383f6" Nov 03 22:10:56.944899 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:10:56.944863 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:11:01.248671 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:01.248633 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:11:01.585715 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:01.585635 5973 scope.go:122] "RemoveContainer" containerID="b34b1f3618c7a9f84c7708b5e45b838d01b4ab3489ad1ee06cfe82516a3d0f5e" Nov 03 22:11:01.594831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:01.594809 5973 scope.go:122] "RemoveContainer" containerID="6638a8ac7afef00ddc111cbb74bdd93bc91a801a7294d6123ac87d3c2e988aff" Nov 03 22:11:01.602999 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:01.602981 5973 scope.go:122] "RemoveContainer" containerID="e01449a47b21b4afdbd2a8880bebcada3805b34f2804d06f81bd0bc5ae7e1c34" Nov 03 22:11:06.353803 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:06.353773 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:11:10.979290 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.979261 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j"] Nov 03 22:11:10.979851 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.979831 5973 cpu_manager.go:414] "containerMap: removing container" podUID="39be2535-24b7-49ee-b6ed-913b2e6c1c7a" containerName="revoke-keys" Nov 03 22:11:10.979930 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.979853 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39be2535-24b7-49ee-b6ed-913b2e6c1c7a" containerName="revoke-keys" Nov 03 22:11:10.979987 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.979977 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="39be2535-24b7-49ee-b6ed-913b2e6c1c7a" containerName="revoke-keys" Nov 03 22:11:10.984806 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.984785 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:10.989578 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.989554 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:11:10.989729 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.989620 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:11:10.990593 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.990572 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:11:10.990702 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.990637 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:11:10.993802 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:10.993774 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j"] Nov 03 22:11:11.129876 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.129842 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72hf\" (UniqueName: \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.130051 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.129883 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.130051 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.129900 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.230913 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.230845 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72hf\" (UniqueName: \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.230913 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.230884 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.230913 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.230905 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.231563 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.231543 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.233338 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.233313 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.243410 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.243390 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72hf\" (UniqueName: \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\") pod \"maas-api-revoke-sub-f03629db-xcp4j\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.299684 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.299659 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:11.577895 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:11.577752 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j"] Nov 03 22:11:11.580488 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:11:11.580443 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac9356_61a4_45c9_a8ff_0805e73d1154.slice/crio-fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761 WatchSource:0}: Error finding container fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761: Status 404 returned error can't find the container with id fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761 Nov 03 22:11:12.157531 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:12.157500 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" event={"ID":"caac9356-61a4-45c9-a8ff-0805e73d1154","Type":"ContainerStarted","Data":"d751f9d0e4e0c560133f5fc0b9eb98e82a5dd59d428b510580ceaa89a0cbc0b0"} Nov 03 22:11:12.157936 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:12.157559 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" event={"ID":"caac9356-61a4-45c9-a8ff-0805e73d1154","Type":"ContainerStarted","Data":"fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761"} Nov 03 22:11:12.176977 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:12.176933 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" podStartSLOduration=2.176920536 podStartE2EDuration="2.176920536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:12.174681235 +0000 UTC m=+1511.949531080" watchObservedRunningTime="2026-11-03 22:11:12.176920536 +0000 UTC m=+1511.951770406" Nov 03 22:11:13.163907 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:13.163868 5973 generic.go:356] "Generic (PLEG): container finished" podID="caac9356-61a4-45c9-a8ff-0805e73d1154" containerID="d751f9d0e4e0c560133f5fc0b9eb98e82a5dd59d428b510580ceaa89a0cbc0b0" exitCode=0 Nov 03 22:11:13.164391 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:13.163956 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" event={"ID":"caac9356-61a4-45c9-a8ff-0805e73d1154","Type":"ContainerDied","Data":"d751f9d0e4e0c560133f5fc0b9eb98e82a5dd59d428b510580ceaa89a0cbc0b0"} Nov 03 22:11:14.308779 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.308758 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:14.398398 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:11:14.398364 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:11:14.457826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.457745 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\") pod \"caac9356-61a4-45c9-a8ff-0805e73d1154\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " Nov 03 22:11:14.457826 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.457821 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\") pod \"caac9356-61a4-45c9-a8ff-0805e73d1154\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " Nov 03 22:11:14.458069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.457849 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\" (UniqueName: \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\") pod \"caac9356-61a4-45c9-a8ff-0805e73d1154\" (UID: \"caac9356-61a4-45c9-a8ff-0805e73d1154\") " Nov 03 22:11:14.459096 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.458762 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca" pod "caac9356-61a4-45c9-a8ff-0805e73d1154" (UID: "caac9356-61a4-45c9-a8ff-0805e73d1154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:11:14.460318 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.460287 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca" pod "caac9356-61a4-45c9-a8ff-0805e73d1154" (UID: "caac9356-61a4-45c9-a8ff-0805e73d1154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:14.461233 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.461210 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf" pod "caac9356-61a4-45c9-a8ff-0805e73d1154" (UID: "caac9356-61a4-45c9-a8ff-0805e73d1154"). InnerVolumeSpecName "kube-api-access-r72hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:14.559586 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.559553 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:14.559586 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.559585 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac9356-61a4-45c9-a8ff-0805e73d1154-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:14.559782 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:14.559597 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72hf\" (UniqueName: \"kubernetes.io/projected/caac9356-61a4-45c9-a8ff-0805e73d1154-kube-api-access-r72hf\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:15.176827 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:15.176798 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" Nov 03 22:11:15.176997 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:15.176838 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xcp4j" event={"ID":"caac9356-61a4-45c9-a8ff-0805e73d1154","Type":"ContainerDied","Data":"fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761"} Nov 03 22:11:15.176997 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:15.176898 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc9f976f543feab3f5be7d701fce370fe2665d0ec6134cd4897270e88ebd6761" Nov 03 22:11:19.022311 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:19.022280 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k"] Nov 03 22:11:19.024868 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:19.024846 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qgf7k"] Nov 03 22:11:20.345894 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:20.345860 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453358f7-57fd-4057-ad7f-05deba163909" path="/var/lib/kubelet/pods/453358f7-57fd-4057-ad7f-05deba163909/volumes" Nov 03 22:11:21.646297 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:21.646249 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:11:26.849890 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.849859 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr"] Nov 03 22:11:26.850314 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.850269 5973 cpu_manager.go:414] "containerMap: removing container" podUID="caac9356-61a4-45c9-a8ff-0805e73d1154" containerName="revoke-keys" Nov 03 22:11:26.850314 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.850280 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caac9356-61a4-45c9-a8ff-0805e73d1154" containerName="revoke-keys" Nov 03 22:11:26.850401 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.850357 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="caac9356-61a4-45c9-a8ff-0805e73d1154" containerName="revoke-keys" Nov 03 22:11:26.853532 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.853517 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.859668 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.859642 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:11:26.859807 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.859740 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:11:26.860794 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.860774 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:11:26.860939 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.860874 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:11:26.863898 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.863877 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr"] Nov 03 22:11:26.894949 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.894927 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.895075 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.894975 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.895138 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.895116 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrqq\" (UniqueName: \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.996582 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.996548 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.996787 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.996610 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.996787 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.996653 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrqq\" (UniqueName: \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.997226 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.997197 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:26.999209 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:26.999181 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:27.009310 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:27.009289 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrqq\" (UniqueName: \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\") pod \"maas-api-revoke-sub-0165d7d2-kg9gr\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:27.166770 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:27.166741 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:27.442957 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:27.442933 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr"] Nov 03 22:11:27.444358 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:11:27.444330 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5414942_d59c_4330_8bfe_a60565947cd7.slice/crio-1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91 WatchSource:0}: Error finding container 1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91: Status 404 returned error can't find the container with id 1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91 Nov 03 22:11:28.254259 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:28.254213 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" event={"ID":"e5414942-d59c-4330-8bfe-a60565947cd7","Type":"ContainerStarted","Data":"a644e226354e7e511f0abe55164c496a8df258b58f3b3aeffdbbe032d9197f5c"} Nov 03 22:11:28.254742 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:28.254276 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" event={"ID":"e5414942-d59c-4330-8bfe-a60565947cd7","Type":"ContainerStarted","Data":"1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91"} Nov 03 22:11:28.275687 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:28.275643 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" podStartSLOduration=2.275631477 podStartE2EDuration="2.275631477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:28.273647184 +0000 UTC m=+1528.048497032" watchObservedRunningTime="2026-11-03 22:11:28.275631477 +0000 UTC m=+1528.050481324" Nov 03 22:11:29.268068 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:29.268036 5973 generic.go:356] "Generic (PLEG): container finished" podID="e5414942-d59c-4330-8bfe-a60565947cd7" containerID="a644e226354e7e511f0abe55164c496a8df258b58f3b3aeffdbbe032d9197f5c" exitCode=0 Nov 03 22:11:29.268450 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:29.268094 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" event={"ID":"e5414942-d59c-4330-8bfe-a60565947cd7","Type":"ContainerDied","Data":"a644e226354e7e511f0abe55164c496a8df258b58f3b3aeffdbbe032d9197f5c"} Nov 03 22:11:30.395428 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.395405 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:30.528000 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.527921 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\") pod \"e5414942-d59c-4330-8bfe-a60565947cd7\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " Nov 03 22:11:30.528000 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.527959 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\") pod \"e5414942-d59c-4330-8bfe-a60565947cd7\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " Nov 03 22:11:30.528198 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.528050 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\" (UniqueName: \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\") pod \"e5414942-d59c-4330-8bfe-a60565947cd7\" (UID: \"e5414942-d59c-4330-8bfe-a60565947cd7\") " Nov 03 22:11:30.528294 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.528272 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca" pod "e5414942-d59c-4330-8bfe-a60565947cd7" (UID: "e5414942-d59c-4330-8bfe-a60565947cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:11:30.530266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.530241 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq" pod "e5414942-d59c-4330-8bfe-a60565947cd7" (UID: "e5414942-d59c-4330-8bfe-a60565947cd7"). InnerVolumeSpecName "kube-api-access-2vrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:30.530266 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.530242 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca" pod "e5414942-d59c-4330-8bfe-a60565947cd7" (UID: "e5414942-d59c-4330-8bfe-a60565947cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:30.629577 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.629549 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrqq\" (UniqueName: \"kubernetes.io/projected/e5414942-d59c-4330-8bfe-a60565947cd7-kube-api-access-2vrqq\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:30.629577 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.629572 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:30.629577 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:30.629582 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5414942-d59c-4330-8bfe-a60565947cd7-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:31.279332 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:31.279303 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" Nov 03 22:11:31.279526 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:31.279339 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kg9gr" event={"ID":"e5414942-d59c-4330-8bfe-a60565947cd7","Type":"ContainerDied","Data":"1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91"} Nov 03 22:11:31.279526 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:31.279390 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cfef961bc302958707d242d1c581fb85128a2d15225928bd730d1880056df91" Nov 03 22:11:37.650738 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.650693 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs"] Nov 03 22:11:37.651294 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.651273 5973 cpu_manager.go:414] "containerMap: removing container" podUID="e5414942-d59c-4330-8bfe-a60565947cd7" containerName="revoke-keys" Nov 03 22:11:37.651371 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.651298 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5414942-d59c-4330-8bfe-a60565947cd7" containerName="revoke-keys" Nov 03 22:11:37.651427 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.651390 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5414942-d59c-4330-8bfe-a60565947cd7" containerName="revoke-keys" Nov 03 22:11:37.654755 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.654734 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.661847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.661017 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:11:37.661847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.661168 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5n7j7\"" Nov 03 22:11:37.661847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.661232 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:11:37.661847 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.661293 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:11:37.665195 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.665163 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs"] Nov 03 22:11:37.797965 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.797936 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.798135 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.797984 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8cv\" (UniqueName: \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.798135 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.798097 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.898593 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.898564 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.898748 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.898610 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8cv\" (UniqueName: \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.898748 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.898650 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.899158 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.899138 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.901048 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.901005 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.911095 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.911071 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8cv\" (UniqueName: \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\") pod \"maas-api-revoke-sub-6b19338f-pfmjs\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:37.968485 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:37.968444 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:38.245167 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:38.245135 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs"] Nov 03 22:11:38.246845 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:11:38.246814 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e55186_4c8f_4b90_a930_ad2925e93f0c.slice/crio-49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b WatchSource:0}: Error finding container 49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b: Status 404 returned error can't find the container with id 49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b Nov 03 22:11:38.317406 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:38.317376 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" event={"ID":"53e55186-4c8f-4b90-a930-ad2925e93f0c","Type":"ContainerStarted","Data":"49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b"} Nov 03 22:11:39.324162 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:39.324131 5973 generic.go:356] "Generic (PLEG): container finished" podID="53e55186-4c8f-4b90-a930-ad2925e93f0c" containerID="9c184219e76c9fd78a8a4811baf963e82772f81346eb4550182f1e20a3005462" exitCode=0 Nov 03 22:11:39.324593 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:39.324221 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" event={"ID":"53e55186-4c8f-4b90-a930-ad2925e93f0c","Type":"ContainerDied","Data":"9c184219e76c9fd78a8a4811baf963e82772f81346eb4550182f1e20a3005462"} Nov 03 22:11:40.456646 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.456624 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:40.620534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.620419 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\") pod \"53e55186-4c8f-4b90-a930-ad2925e93f0c\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " Nov 03 22:11:40.620534 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.620517 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\" (UniqueName: \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\") pod \"53e55186-4c8f-4b90-a930-ad2925e93f0c\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " Nov 03 22:11:40.620818 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.620669 5973 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\") pod \"53e55186-4c8f-4b90-a930-ad2925e93f0c\" (UID: \"53e55186-4c8f-4b90-a930-ad2925e93f0c\") " Nov 03 22:11:40.621053 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.621025 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca" pod "53e55186-4c8f-4b90-a930-ad2925e93f0c" (UID: "53e55186-4c8f-4b90-a930-ad2925e93f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:11:40.622878 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.622856 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca" pod "53e55186-4c8f-4b90-a930-ad2925e93f0c" (UID: "53e55186-4c8f-4b90-a930-ad2925e93f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:40.622878 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.622861 5973 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv" pod "53e55186-4c8f-4b90-a930-ad2925e93f0c" (UID: "53e55186-4c8f-4b90-a930-ad2925e93f0c"). InnerVolumeSpecName "kube-api-access-9g8cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:40.721635 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.721596 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-openshift-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:40.721635 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.721631 5973 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e55186-4c8f-4b90-a930-ad2925e93f0c-maas-api-xks-ca\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:40.721822 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:40.721645 5973 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8cv\" (UniqueName: \"kubernetes.io/projected/53e55186-4c8f-4b90-a930-ad2925e93f0c-kube-api-access-9g8cv\") on node \"ip-10-0-131-89.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:41.336435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:41.336402 5973 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" Nov 03 22:11:41.336435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:41.336419 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pfmjs" event={"ID":"53e55186-4c8f-4b90-a930-ad2925e93f0c","Type":"ContainerDied","Data":"49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b"} Nov 03 22:11:41.336705 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:41.336500 5973 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e88ae876ba61af79c5a0ee741da569248ba6dee4e4bcc95a7b4b9ded1dae6b" Nov 03 22:11:44.398683 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:11:44.398643 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:11:48.361436 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:48.361403 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qq89p"] Nov 03 22:11:48.944385 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:48.943570 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564bd7d5c5-6g6pd_a1cd5aaf-0692-4257-8095-7882434c9927/authorino/0.log" Nov 03 22:11:58.963133 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:11:58.963097 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f567997d9-rt9jg_5bf69925-4d42-4335-b175-e9944add6bcb/manager/0.log" Nov 03 22:12:00.440445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:00.440415 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-j5t6s_53306a2c-5996-4ad9-bd5a-631945b18bec/manager/0.log" Nov 03 22:12:00.875118 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:00.875047 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/copy-manifests/0.log" Nov 03 22:12:00.898126 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:00.898104 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/manager/0.log" Nov 03 22:12:01.639840 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:01.639807 5973 scope.go:122] "RemoveContainer" containerID="6c543183d6ce0cdd6cf38d36069ad180ab07971f66e28771873d439173fcfe6a" Nov 03 22:12:01.648964 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:01.648942 5973 scope.go:122] "RemoveContainer" containerID="76db18a41037ab45ccd346800bb1317fa5c348d924a5d17319ba5f278ecaa699" Nov 03 22:12:02.440046 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:02.440016 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kg9gr_e5414942-d59c-4330-8bfe-a60565947cd7/revoke-keys/0.log" Nov 03 22:12:02.606734 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:02.606705 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-76xdl_166b46e6-7247-467e-80fd-652e03ec045b/revoke-keys/0.log" Nov 03 22:12:02.783821 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:02.783745 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8ddhd_08a4f860-3c4a-4ce7-8357-542380fa1705/revoke-keys/0.log" Nov 03 22:12:02.954757 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:02.954723 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-529a373e-6w74m_4f780388-390b-41bc-a296-9c4e9c13390d/revoke-keys/0.log" Nov 03 22:12:03.122595 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:03.122516 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pfmjs_53e55186-4c8f-4b90-a930-ad2925e93f0c/revoke-keys/0.log" Nov 03 22:12:03.303270 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:03.303244 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ttkw9_815b129a-64f1-4520-8039-97e095097a4a/revoke-keys/0.log" Nov 03 22:12:03.480742 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:03.480701 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-nm2cz_29658059-0d2b-486b-84e8-506b4985c819/revoke-keys/0.log" Nov 03 22:12:03.654008 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:03.653979 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee5f23f9-l96tb_39be2535-24b7-49ee-b6ed-913b2e6c1c7a/revoke-keys/0.log" Nov 03 22:12:03.820435 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:03.820352 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xcp4j_caac9356-61a4-45c9-a8ff-0805e73d1154/revoke-keys/0.log" Nov 03 22:12:05.465410 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:05.465378 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564bd7d5c5-6g6pd_a1cd5aaf-0692-4257-8095-7882434c9927/authorino/0.log" Nov 03 22:12:06.045641 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:06.045612 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9qw28_fb871a0f-2db0-483b-af0f-818f79cd9b31/kuadrant-console-plugin/0.log" Nov 03 22:12:06.458891 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:06.458858 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qq89p_ce277295-b834-4149-9c99-e921a9277b32/limitador/0.log" Nov 03 22:12:07.781432 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:07.781397 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65998dd86f-c9x59_95bda906-6e41-4c07-bf43-ea992687d417/kube-auth-proxy/0.log" Nov 03 22:12:08.488679 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:08.488653 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-5cffb_696e0ecd-a455-4af2-bdb4-9932fab6d32d/istio-proxy/0.log" Nov 03 22:12:08.855265 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:08.855180 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-7z9gp_3799a49d-c03b-43d4-b7ba-c2155e797733/istio-proxy/0.log" Nov 03 22:12:09.043424 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:09.043391 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bkhgj_b5b28807-4eee-4373-8d54-526f477ae078/istio-proxy/0.log" Nov 03 22:12:11.574539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:11.574501 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lbjgv_255a3973-a3ab-420c-adc2-6afa00b99c71/main/0.log" Nov 03 22:12:11.930726 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:11.930694 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm_e2a21998-3b3d-4421-a710-bbb050edb7f3/main/0.log" Nov 03 22:12:12.280247 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:12.280133 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw_e47462ee-ac43-4f06-aeda-0dd21997a7be/main/0.log" Nov 03 22:12:13.149069 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:13.149032 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564bd7d5c5-6g6pd_a1cd5aaf-0692-4257-8095-7882434c9927/authorino/0.log" Nov 03 22:12:14.398043 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:12:14.398009 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:12:22.008734 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:22.008684 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f567997d9-rt9jg_5bf69925-4d42-4335-b175-e9944add6bcb/manager/0.log" Nov 03 22:12:23.418668 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:23.418623 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-j5t6s_53306a2c-5996-4ad9-bd5a-631945b18bec/manager/0.log" Nov 03 22:12:23.845477 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:23.845394 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/copy-manifests/0.log" Nov 03 22:12:23.863682 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:23.863653 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/manager/0.log" Nov 03 22:12:25.166999 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:25.166960 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kg9gr_e5414942-d59c-4330-8bfe-a60565947cd7/revoke-keys/0.log" Nov 03 22:12:25.338043 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:25.338013 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-76xdl_166b46e6-7247-467e-80fd-652e03ec045b/revoke-keys/0.log" Nov 03 22:12:25.513651 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:25.513574 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8ddhd_08a4f860-3c4a-4ce7-8357-542380fa1705/revoke-keys/0.log" Nov 03 22:12:25.691023 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:25.690991 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-529a373e-6w74m_4f780388-390b-41bc-a296-9c4e9c13390d/revoke-keys/0.log" Nov 03 22:12:25.868493 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:25.868384 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pfmjs_53e55186-4c8f-4b90-a930-ad2925e93f0c/revoke-keys/0.log" Nov 03 22:12:26.035882 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:26.035852 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ttkw9_815b129a-64f1-4520-8039-97e095097a4a/revoke-keys/0.log" Nov 03 22:12:26.212824 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:26.212793 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-nm2cz_29658059-0d2b-486b-84e8-506b4985c819/revoke-keys/0.log" Nov 03 22:12:26.373766 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:26.373739 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee5f23f9-l96tb_39be2535-24b7-49ee-b6ed-913b2e6c1c7a/revoke-keys/0.log" Nov 03 22:12:26.540548 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:26.540431 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xcp4j_caac9356-61a4-45c9-a8ff-0805e73d1154/revoke-keys/0.log" Nov 03 22:12:27.793709 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:27.793673 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564bd7d5c5-6g6pd_a1cd5aaf-0692-4257-8095-7882434c9927/authorino/0.log" Nov 03 22:12:28.369122 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:28.369089 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9qw28_fb871a0f-2db0-483b-af0f-818f79cd9b31/kuadrant-console-plugin/0.log" Nov 03 22:12:28.761953 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:28.761917 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qq89p_ce277295-b834-4149-9c99-e921a9277b32/limitador/0.log" Nov 03 22:12:29.859991 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:29.859959 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65998dd86f-c9x59_95bda906-6e41-4c07-bf43-ea992687d417/kube-auth-proxy/0.log" Nov 03 22:12:30.584168 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:30.584144 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-5cffb_696e0ecd-a455-4af2-bdb4-9932fab6d32d/istio-proxy/0.log" Nov 03 22:12:30.955186 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:30.955154 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-7z9gp_3799a49d-c03b-43d4-b7ba-c2155e797733/istio-proxy/0.log" Nov 03 22:12:31.154700 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:31.154665 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bkhgj_b5b28807-4eee-4373-8d54-526f477ae078/istio-proxy/0.log" Nov 03 22:12:33.241902 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:33.241866 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lbjgv_255a3973-a3ab-420c-adc2-6afa00b99c71/main/0.log" Nov 03 22:12:33.608795 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:33.608707 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8l58gm_e2a21998-3b3d-4421-a710-bbb050edb7f3/main/0.log" Nov 03 22:12:33.940953 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:12:33.940896 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7wcdw_e47462ee-ac43-4f06-aeda-0dd21997a7be/main/0.log" Nov 03 22:12:44.399215 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:12:44.399177 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:13:00.023881 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:00.023841 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz"] Nov 03 22:13:00.026379 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:00.026351 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nm2cz"] Nov 03 22:13:00.345545 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:00.345424 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29658059-0d2b-486b-84e8-506b4985c819" path="/var/lib/kubelet/pods/29658059-0d2b-486b-84e8-506b4985c819/volumes" Nov 03 22:13:01.673752 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:01.673719 5973 scope.go:122] "RemoveContainer" containerID="782d401cec88e47e8479c69c8d6b0f91b5dd59b891d0b8255aacc916de6eba0f" Nov 03 22:13:14.399041 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:13:14.398964 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:13:17.023249 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:17.023206 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9"] Nov 03 22:13:17.026015 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:17.025987 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ttkw9"] Nov 03 22:13:18.344164 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:18.344130 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815b129a-64f1-4520-8039-97e095097a4a" path="/var/lib/kubelet/pods/815b129a-64f1-4520-8039-97e095097a4a/volumes" Nov 03 22:13:44.398936 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:13:44.398894 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:13:58.070535 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.070501 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-69rm6/must-gather-rfsqs"] Nov 03 22:13:58.070944 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.070927 5973 cpu_manager.go:414] "containerMap: removing container" podUID="53e55186-4c8f-4b90-a930-ad2925e93f0c" containerName="revoke-keys" Nov 03 22:13:58.070993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.070946 5973 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e55186-4c8f-4b90-a930-ad2925e93f0c" containerName="revoke-keys" Nov 03 22:13:58.071034 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.071000 5973 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e55186-4c8f-4b90-a930-ad2925e93f0c" containerName="revoke-keys" Nov 03 22:13:58.074116 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.074100 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.078822 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.078800 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-69rm6\"/\"openshift-service-ca.crt\"" Nov 03 22:13:58.078926 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.078859 5973 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-69rm6\"/\"kube-root-ca.crt\"" Nov 03 22:13:58.078926 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.078916 5973 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-69rm6\"/\"default-dockercfg-wv8q7\"" Nov 03 22:13:58.092790 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.092771 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-69rm6/must-gather-rfsqs"] Nov 03 22:13:58.212251 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.212224 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54p4\" (UniqueName: \"kubernetes.io/projected/221be16d-1938-4e9d-adf9-50d46b538dc5-kube-api-access-n54p4\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.212427 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.212256 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/221be16d-1938-4e9d-adf9-50d46b538dc5-must-gather-output\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.312993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.312960 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p4\" (UniqueName: \"kubernetes.io/projected/221be16d-1938-4e9d-adf9-50d46b538dc5-kube-api-access-n54p4\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.312993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.312996 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/221be16d-1938-4e9d-adf9-50d46b538dc5-must-gather-output\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.313356 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.313339 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/221be16d-1938-4e9d-adf9-50d46b538dc5-must-gather-output\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.327850 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.327789 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54p4\" (UniqueName: \"kubernetes.io/projected/221be16d-1938-4e9d-adf9-50d46b538dc5-kube-api-access-n54p4\") pod \"must-gather-rfsqs\" (UID: \"221be16d-1938-4e9d-adf9-50d46b538dc5\") " pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.388395 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.388372 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-69rm6/must-gather-rfsqs" Nov 03 22:13:58.662993 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.662966 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-69rm6/must-gather-rfsqs"] Nov 03 22:13:58.664559 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:13:58.664531 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod221be16d_1938_4e9d_adf9_50d46b538dc5.slice/crio-51b4fd85831f0900ef2ead377a97b0b430cd5968ec1f3f9ce25fecb5df82c967 WatchSource:0}: Error finding container 51b4fd85831f0900ef2ead377a97b0b430cd5968ec1f3f9ce25fecb5df82c967: Status 404 returned error can't find the container with id 51b4fd85831f0900ef2ead377a97b0b430cd5968ec1f3f9ce25fecb5df82c967 Nov 03 22:13:58.666831 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:58.666810 5973 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:13:59.116985 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:13:59.116909 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-69rm6/must-gather-rfsqs" event={"ID":"221be16d-1938-4e9d-adf9-50d46b538dc5","Type":"ContainerStarted","Data":"51b4fd85831f0900ef2ead377a97b0b430cd5968ec1f3f9ce25fecb5df82c967"} Nov 03 22:14:00.126698 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:00.126659 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-69rm6/must-gather-rfsqs" event={"ID":"221be16d-1938-4e9d-adf9-50d46b538dc5","Type":"ContainerStarted","Data":"01e62afc6575cbac25a2ddf728d795277c3c57dac0efb01f8cda30ac1140ad22"} Nov 03 22:14:00.127195 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:00.126719 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-69rm6/must-gather-rfsqs" event={"ID":"221be16d-1938-4e9d-adf9-50d46b538dc5","Type":"ContainerStarted","Data":"f2819294a4b9cf89e4fe84c009f04fb76202449de65ab50f22556826af3c1ce1"} Nov 03 22:14:00.150341 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:00.149094 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-69rm6/must-gather-rfsqs" podStartSLOduration=1.40978949 podStartE2EDuration="2.149073096s" totalImagesPullingTime="739.283606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:00.145659467 +0000 UTC m=+1679.920509315" watchObservedRunningTime="2026-11-03 22:14:00.149073096 +0000 UTC m=+1679.923922944" Nov 03 22:14:01.155255 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:01.155223 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cw9gc_82debff8-da8f-4dfa-980b-374e735bd8c4/global-pull-secret-syncer/0.log" Nov 03 22:14:01.444299 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:01.444216 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hx8cd_feec6a26-5dc3-4688-829d-f5316d6bda7d/konnectivity-agent/0.log" Nov 03 22:14:01.565445 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:01.565407 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-89.us-east-2.compute.internal_a387975d67fca887e2f3c912beb1760d/haproxy/0.log" Nov 03 22:14:01.708754 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:01.708663 5973 scope.go:122] "RemoveContainer" containerID="e294d55b4d2c366186274267041c6d24094592e0628b5f0c8671e54be0f6458a" Nov 03 22:14:06.030503 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:06.030444 5973 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m"] Nov 03 22:14:06.031206 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:06.031185 5973 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529a373e-6w74m"] Nov 03 22:14:06.348528 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:06.348196 5973 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f780388-390b-41bc-a296-9c4e9c13390d" path="/var/lib/kubelet/pods/4f780388-390b-41bc-a296-9c4e9c13390d/volumes" Nov 03 22:14:08.513482 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:08.510508 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564bd7d5c5-6g6pd_a1cd5aaf-0692-4257-8095-7882434c9927/authorino/0.log" Nov 03 22:14:08.607216 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:08.605026 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9qw28_fb871a0f-2db0-483b-af0f-818f79cd9b31/kuadrant-console-plugin/0.log" Nov 03 22:14:08.708483 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:08.707816 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qq89p_ce277295-b834-4149-9c99-e921a9277b32/limitador/0.log" Nov 03 22:14:10.060876 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.060851 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/alertmanager/0.log" Nov 03 22:14:10.089210 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.089178 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/config-reloader/0.log" Nov 03 22:14:10.118888 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.118845 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/kube-rbac-proxy-web/0.log" Nov 03 22:14:10.146413 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.146375 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/kube-rbac-proxy/0.log" Nov 03 22:14:10.177490 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.176400 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/kube-rbac-proxy-metric/0.log" Nov 03 22:14:10.201948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.201917 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/prom-label-proxy/0.log" Nov 03 22:14:10.228355 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.228326 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a3b76f3-a06f-42d5-b2e6-d50d90ef4731/init-config-reloader/0.log" Nov 03 22:14:10.300521 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.300491 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h8hsl_24414154-24ae-4506-bb36-49d64ba5fcef/kube-state-metrics/0.log" Nov 03 22:14:10.320225 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.320146 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h8hsl_24414154-24ae-4506-bb36-49d64ba5fcef/kube-rbac-proxy-main/0.log" Nov 03 22:14:10.363211 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.363164 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h8hsl_24414154-24ae-4506-bb36-49d64ba5fcef/kube-rbac-proxy-self/0.log" Nov 03 22:14:10.391817 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.391789 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79b56c756-lg8dh_cfb89632-d017-4912-9bd3-ae92a2e187fe/metrics-server/0.log" Nov 03 22:14:10.552603 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.552401 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnlbr_33267669-fbef-4d65-a6f3-23b24f74ca7f/node-exporter/0.log" Nov 03 22:14:10.576168 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.576088 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnlbr_33267669-fbef-4d65-a6f3-23b24f74ca7f/kube-rbac-proxy/0.log" Nov 03 22:14:10.601050 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.601016 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnlbr_33267669-fbef-4d65-a6f3-23b24f74ca7f/init-textfile/0.log" Nov 03 22:14:10.725478 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.724871 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lg4xh_aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/kube-rbac-proxy-main/0.log" Nov 03 22:14:10.750211 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.750177 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lg4xh_aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/kube-rbac-proxy-self/0.log" Nov 03 22:14:10.776070 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.776026 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lg4xh_aaaf2d59-8e11-4be0-a9c6-4dd5efb74d85/openshift-state-metrics/0.log" Nov 03 22:14:10.995481 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:10.995262 5973 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj"] Nov 03 22:14:11.004170 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.002840 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.006666 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.006636 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj"] Nov 03 22:14:11.095089 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.095059 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbvh\" (UniqueName: \"kubernetes.io/projected/bfd58db8-c82e-407d-8ac3-2a42f158cca6-kube-api-access-vsbvh\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.095089 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.095096 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-proc\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.095650 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.095113 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-podres\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.095650 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.095198 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-lib-modules\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.095650 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.095337 5973 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-sys\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.137491 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.137366 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f954c659-h7nqb_5baffe5c-d923-43b8-a3d4-7d1edf10e744/telemeter-client/0.log" Nov 03 22:14:11.166476 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.166426 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f954c659-h7nqb_5baffe5c-d923-43b8-a3d4-7d1edf10e744/reload/0.log" Nov 03 22:14:11.191158 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.191132 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f954c659-h7nqb_5baffe5c-d923-43b8-a3d4-7d1edf10e744/kube-rbac-proxy/0.log" Nov 03 22:14:11.196628 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196594 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-proc\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196778 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196640 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-podres\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196778 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196681 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-lib-modules\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196778 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196729 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-proc\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196802 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-podres\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196811 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-lib-modules\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196832 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-sys\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196931 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfd58db8-c82e-407d-8ac3-2a42f158cca6-sys\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.196948 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.196936 5973 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbvh\" (UniqueName: \"kubernetes.io/projected/bfd58db8-c82e-407d-8ac3-2a42f158cca6-kube-api-access-vsbvh\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.209016 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.208989 5973 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbvh\" (UniqueName: \"kubernetes.io/projected/bfd58db8-c82e-407d-8ac3-2a42f158cca6-kube-api-access-vsbvh\") pod \"perf-node-gather-daemonset-n5qkj\" (UID: \"bfd58db8-c82e-407d-8ac3-2a42f158cca6\") " pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.323633 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.323542 5973 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:11.612739 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:11.612712 5973 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj"] Nov 03 22:14:11.613348 ip-10-0-131-89 kubenswrapper[5973]: W1103 22:14:11.613323 5973 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfd58db8_c82e_407d_8ac3_2a42f158cca6.slice/crio-b37c4ef0bdcd2e696ccb3134c2d02dfedda6cc0e1d55feff5ed80a6e6e8c55bc WatchSource:0}: Error finding container b37c4ef0bdcd2e696ccb3134c2d02dfedda6cc0e1d55feff5ed80a6e6e8c55bc: Status 404 returned error can't find the container with id b37c4ef0bdcd2e696ccb3134c2d02dfedda6cc0e1d55feff5ed80a6e6e8c55bc Nov 03 22:14:12.263151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:12.260068 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" event={"ID":"bfd58db8-c82e-407d-8ac3-2a42f158cca6","Type":"ContainerStarted","Data":"a26315f0f35d8a2a01472e600006bd11ccd605afc1f783051ee6e7c4593d6c3c"} Nov 03 22:14:12.263151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:12.260122 5973 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" event={"ID":"bfd58db8-c82e-407d-8ac3-2a42f158cca6","Type":"ContainerStarted","Data":"b37c4ef0bdcd2e696ccb3134c2d02dfedda6cc0e1d55feff5ed80a6e6e8c55bc"} Nov 03 22:14:12.263151 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:12.260845 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:12.279886 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:12.279826 5973 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" podStartSLOduration=2.279810961 podStartE2EDuration="2.279810961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:12.278787927 +0000 UTC m=+1692.053637774" watchObservedRunningTime="2026-11-03 22:14:12.279810961 +0000 UTC m=+1692.054660824" Nov 03 22:14:13.671277 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:13.671247 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-699c6fd85b-jjs28_1b294cd0-8239-4ec3-ad32-adff86b0f779/console/0.log" Nov 03 22:14:14.422485 ip-10-0-131-89 kubenswrapper[5973]: E1103 22:14:14.421073 5973 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-snnch" containerName="extractor" Nov 03 22:14:15.510407 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:15.510377 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hvnmj_38585226-48bc-4637-88aa-ff1cf9b9fd78/dns/0.log" Nov 03 22:14:15.534272 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:15.534244 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hvnmj_38585226-48bc-4637-88aa-ff1cf9b9fd78/kube-rbac-proxy/0.log" Nov 03 22:14:15.684413 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:15.684381 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnwgp_a9135178-a435-448e-b425-e821a0470160/dns-node-resolver/0.log" Nov 03 22:14:16.445088 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:16.445052 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t265q_0c9f0f1e-3e27-4f6a-aff3-a621b458030b/node-ca/0.log" Nov 03 22:14:18.196031 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:18.195995 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65998dd86f-c9x59_95bda906-6e41-4c07-bf43-ea992687d417/kube-auth-proxy/0.log" Nov 03 22:14:18.326539 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:18.326511 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-5cffb_696e0ecd-a455-4af2-bdb4-9932fab6d32d/istio-proxy/0.log" Nov 03 22:14:18.395157 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:18.395125 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-7z9gp_3799a49d-c03b-43d4-b7ba-c2155e797733/istio-proxy/0.log" Nov 03 22:14:18.427222 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:18.427166 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bkhgj_b5b28807-4eee-4373-8d54-526f477ae078/istio-proxy/0.log" Nov 03 22:14:19.290322 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:19.290293 5973 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-69rm6/perf-node-gather-daemonset-n5qkj" Nov 03 22:14:19.642947 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:19.642883 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8j8zm_12498367-0192-4402-bb17-bfe4f3432d3a/serve-healthcheck-canary/0.log" Nov 03 22:14:20.369206 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:20.369174 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-snnch_d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/kube-rbac-proxy/0.log" Nov 03 22:14:20.394117 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:20.394087 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-snnch_d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/exporter/0.log" Nov 03 22:14:20.421884 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:20.420390 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-snnch_d5a9e1cd-c054-453d-bd1f-fdaa0a35e700/extractor/0.log" Nov 03 22:14:22.180898 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:22.180855 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65998dd86f-c9x59_95bda906-6e41-4c07-bf43-ea992687d417/kube-auth-proxy/0.log" Nov 03 22:14:22.255071 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:22.255039 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-5cffb_696e0ecd-a455-4af2-bdb4-9932fab6d32d/istio-proxy/0.log" Nov 03 22:14:22.294304 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:22.294275 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-7z9gp_3799a49d-c03b-43d4-b7ba-c2155e797733/istio-proxy/0.log" Nov 03 22:14:22.310371 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:22.310341 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bkhgj_b5b28807-4eee-4373-8d54-526f477ae078/istio-proxy/0.log" Nov 03 22:14:23.435144 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:23.435111 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f567997d9-rt9jg_5bf69925-4d42-4335-b175-e9944add6bcb/manager/0.log" Nov 03 22:14:23.964625 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:23.964589 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-j5t6s_53306a2c-5996-4ad9-bd5a-631945b18bec/manager/0.log" Nov 03 22:14:24.067648 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.067621 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/manager/0.log" Nov 03 22:14:24.093654 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.093628 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77f6458f94-p87pm_799bbffe-80a5-41be-a49e-8dd63c8bfb18/copy-manifests/0.log" Nov 03 22:14:24.565978 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.565944 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65998dd86f-c9x59_95bda906-6e41-4c07-bf43-ea992687d417/kube-auth-proxy/0.log" Nov 03 22:14:24.637809 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.637768 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-5cffb_696e0ecd-a455-4af2-bdb4-9932fab6d32d/istio-proxy/0.log" Nov 03 22:14:24.674204 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.674173 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-7z9gp_3799a49d-c03b-43d4-b7ba-c2155e797733/istio-proxy/0.log" Nov 03 22:14:24.690789 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:24.690752 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-bkhgj_b5b28807-4eee-4373-8d54-526f477ae078/istio-proxy/0.log" Nov 03 22:14:25.618828 ip-10-0-131-89 kubenswrapper[5973]: I1103 22:14:25.618802 5973 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c6f488f88-q4rdn_22b5f707-a41f-43b1-8ea2-1ad22e4f1934/manager/0.log"